{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_46frs9w",
    "id": "D89AFB24BD6B48F698E0F6DEFD4F4051",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "# 三、多层感知机\n",
    "1. 多层感知机的基本知识\n",
    "2. 使用多层感知机图像分类的从零开始的实现\n",
    "3. 使用pytorch的简洁实现"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_phvkn2u",
    "id": "D0B89D7E2DAB4EFF8267BD113F9DB33D",
    "jupyter": {},
    "mdEditEnable": false,
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## 1、多层感知机的基本知识\n",
    "深度学习主要关注多层模型。在这里，我们将以多层感知机（multilayer perceptron，MLP）为例，介绍多层神经网络的概念。\n",
    "\n",
    "### 隐藏层\n",
    "下图展示了一个多层感知机的神经网络图，它含有一个隐藏层，该层中有5个隐藏单元。\n",
    "\n",
    "![Image Name](./q5ho684jmh.png)\n",
    "\n",
    "### 表达公式\n",
    "具体来说，给定一个小批量样本$\\boldsymbol{X} \\in \\mathbb{R}^{n \\times d}$，其批量大小为$n$，输入个数为$d$。假设多层感知机只有一个隐藏层，其中隐藏单元个数为$h$。记隐藏层的输出（也称为隐藏层变量或隐藏变量）为$\\boldsymbol{H}$，有$\\boldsymbol{H} \\in \\mathbb{R}^{n \\times h}$。因为隐藏层和输出层均是全连接层，可以设隐藏层的权重参数和偏差参数分别为$\\boldsymbol{W}_h \\in \\mathbb{R}^{d \\times h}$和 $\\boldsymbol{b}_h \\in \\mathbb{R}^{1 \\times h}$，输出层的权重和偏差参数分别为$\\boldsymbol{W}_o \\in \\mathbb{R}^{h \\times q}$和$\\boldsymbol{b}_o \\in \\mathbb{R}^{1 \\times q}$。\n",
    "\n",
    "我们先来看一种含单隐藏层的多层感知机的设计。其输出$\\boldsymbol{O} \\in \\mathbb{R}^{n \\times q}$的计算为\n",
    "\n",
    "\n",
    "$$\n",
    " \\begin{aligned} \\boldsymbol{H} &= \\boldsymbol{X} \\boldsymbol{W}_h + \\boldsymbol{b}_h,\\\\ \\boldsymbol{O} &= \\boldsymbol{H} \\boldsymbol{W}_o + \\boldsymbol{b}_o, \\end{aligned}\n",
    "$$\n",
    "\n",
    "\n",
    "也就是将隐藏层的输出直接作为输出层的输入。如果将以上两个式子联立起来，可以得到\n",
    "\n",
    "\n",
    "$$\n",
    " \\boldsymbol{O} = (\\boldsymbol{X} \\boldsymbol{W}_h + \\boldsymbol{b}_h)\\boldsymbol{W}_o + \\boldsymbol{b}_o = \\boldsymbol{X} \\boldsymbol{W}_h\\boldsymbol{W}_o + \\boldsymbol{b}_h \\boldsymbol{W}_o + \\boldsymbol{b}_o. \n",
    "$$\n",
    "\n",
    "\n",
    "从联立后的式子可以看出，虽然神经网络引入了隐藏层，却依然等价于一个单层神经网络：其中输出层权重参数为$\\boldsymbol{W}_h\\boldsymbol{W}_o$，偏差参数为$\\boldsymbol{b}_h \\boldsymbol{W}_o + \\boldsymbol{b}_o$。不难发现，即便再添加更多的隐藏层，以上设计依然只能与仅含输出层的单层神经网络等价。\n",
    "\n",
    "### 激活函数\n",
    "上述问题的根源在于全连接层只是对数据做仿射变换（affine transformation），而多个仿射变换的叠加仍然是一个仿射变换。解决问题的一个方法是引入非线性变换，例如对隐藏变量使用按元素运算的非线性函数进行变换，然后再作为下一个全连接层的输入。这个非线性函数被称为激活函数（activation function）。\n",
    "\n",
    "下面我们介绍几个常用的激活函数：\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_1e20aks",
    "id": "F4D9FF54F80B430691D98382E10172AC",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "#### ReLU函数\n",
    "ReLU（rectified linear unit）函数提供了一个很简单的非线性变换。给定元素$x$，该函数定义为\n",
    "\n",
    "\n",
    "$$\n",
    "\\text{ReLU}(x) = \\max(x, 0).\n",
    "$$\n",
    "\n",
    "\n",
    "可以看出，ReLU函数只保留正数元素，并将负数元素清零。为了直观地观察这一非线性变换，我们先定义一个绘图函数xyplot。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19",
    "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5",
    "graffitiCellId": "id_ucmhsyy",
    "id": "C052F0E71D3D4AF984A86C9CA4175221",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.4.0\n"
     ]
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import torch\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import sys\n",
    "sys.path.append(\"..\")\n",
    "# sys.path.append(\"/home/kesci/input\")\n",
    "import d2lzh1981 as d2l\n",
    "print(torch.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "_cell_guid": "79c7e3d0-c299-4dcb-8224-4455121ee9b0",
    "_uuid": "d629ff2d2480ee46fbb7e2d37f6b5fab8052498a",
    "graffitiCellId": "id_9a5gym1",
    "id": "B51407B8D4D445F793ACED139CA02873",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def xyplot(x_vals, y_vals, name):\n",
    "    # d2l.set_figsize(figsize=(5, 2.5))\n",
    "    plt.plot(x_vals.detach().numpy(), y_vals.detach().numpy())\n",
    "    plt.xlabel('x')\n",
    "    plt.ylabel(name + '(x)')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "graffitiCellId": "id_r7e89xp",
    "id": "070825B6A382411DA5BD7D14E67E8D54",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAEGCAYAAABvtY4XAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAd90lEQVR4nO3dd5xU9dn+8c8tRaQJyAJKkSIIipRlRbBFbLFFkxgLiokxEUNR9DGxRZ/8NCbxsRNbHhI1hQVERE1ssdcouo3eexMWkF522b1/f8ysz4qU3dk9c2bOXO/Xixc7zOx8L2G99+x3zpzL3B0REYmeg8IOICIiwdCAFxGJKA14EZGI0oAXEYkoDXgRkYiqG3aAylq2bOkdO3YMO4aISNrIz89f5+5Ze7svpQZ8x44dycvLCzuGiEjaMLOl+7pPWzQiIhGlAS8iElEa8CIiERXogDezm8xsppnNMLPxZtYgyPVEROT/BDbgzawtcAOQ4+49gTrA5UGtJyIi3xT0Fk1d4BAzqws0BFYFvJ6IiMQFNuDdfSXwILAMWA1scvc393ycmQ01szwzyysuLg4qjohIxglyi6Y5cBHQCTgCaGRmQ/Z8nLuPcfccd8/JytrrufoiIpE1ZdF6nv54MUFcuj3ILZozgcXuXuzupcBk4MQA1xMRSStrt+xk5PhCxn62lB2lZbX+/EEO+GXAADNraGYGnAHMDnA9EZG0sbusnFHji9iys5SnhmTTsH7tX1ggyD34KcAkoACYHl9rTFDriYikk0fenseni9bz24t60r1N00DWCPRaNO7+G+A3Qa4hIpJu3p2zhifeW8hlOe25JKd9YOvonawiIkm04qvt3PTcVHoc3pS7Lzo20LU04EVEkmTX7jJG5BZQXu48dWU2DerVCXS9lLpcsIhIlP3u1dlMXbGJPw3pR8eWjQJfT0fwIiJJ8HLRSv7+6VKuPaUT5/Rsk5Q1NeBFRAK2YO0Wbp88nZwjm3PLOd2Ttq4GvIhIgLaX7GbY2AIOqVeHx6/Ipl6d5I1d7cGLiATE3blj8nQWFG/lH9ecQJtDk3vFdB3Bi4gEZNzny3ipaBU3ndmNk7u2TPr6GvAiIgGYvmITd/9zFqd2y2LkoKNCyaABLyJSyzZtL2VYbj4tG9fn0cv6cNBBFkoO7cGLiNSi8nLn5ueLWLN5J89dN5AWjeqHlkVH8CIitWjMR4t4e/Za7jivB9kdmoeaRQNeRKSWfLZoPQ/8ey7nH3c4V5/YMew4GvAiIrVh7ZadXD++kCNbNOS+i48jVoMRLu3Bi4jU0O6ycm4YX8iWnaX842f9adKgXtiRAA14EZEae/iteXy2aAMPXdI7sPKORARZun20mRVV+rXZzG4Maj0RkTC8M3sNT76/kMH923Nxv3Zhx/mGwI7g3X0u0AfAzOoAK4EXg1pPRCTZlm/Yzn9NnMqxRzTlN98LtrwjEcl6kfUMYKG7L03SeiIigdq1u4wR4wood+fJJJR3JCJZA/5yYPze7jCzoWaWZ2Z5xcXFSYojIlIz974ym2krNvHgJb058rDgyzsSEfiAN7P6wIXA83u7393HuHuOu+dkZWUFHUdEpMZeLlrJPz5bytBTO/PdY5NT3pGIZBzBnwsUuPuaJKwlIhKo+Wti5R3Hd2zOr757dNhx9isZA34w+9ieERFJJ9t27WZYbgEN6ye/vCMRgaYzs4bAWcDkINcREQmau3PHi9NZVLyV0Zf3pXXT5JZ3JCLQNzq5+3bgsCDXEBFJhtwpy3i5aBU3n9WNk45KfnlHIlL75wsRkRQwbcVG7vnXLE47OosRIZV3JEIDXkRkPzZuL2HY2AJaNq7PI5eGV96RCF2LRkRkH8rLnZsnTmXtlp1MvG4gzUMs70iEjuBFRPbhTx8u5J05a7nz/GPoG3J5RyI04EVE9uLThet58N9zuaDX4fx44JFhx0mIBryIyB7Wbo6Vd3Rs2Yj7Lu6VEuUdidAevIhIJbvLyrl+fCFbd5WS+/MTaHxw+o7J9E0uIhKAh96ax5TFG3j40t4c3aZJ2HFqRFs0IiJxb89aw1PvL2Rw/w78MDu1yjsSoQEvIkJFeUdRvLzjmLDj1AoNeBHJeDtLyxiWm48DT13ZLyXLOxKhPXgRyXi/fWUWM1Zu5s8/zqHDYQ3DjlNrdAQvIhntpcKV5E5ZxnWnduasY1qHHadWacCLSMaqKO/o37EFv0zx8o5EaMCLSEbatms3vxibT6OD6/DYFX1TvrwjEdqDF5GM4+7cNnk6i9dtY+zPT0iL8o5EBN3o1MzMJpnZHDObbWYDg1xPRKQqxn62lH9NXcXNZx/NiV3So7wjEUEfwY8G3nD3H5lZfSA6L0+LSFqaunwj97wyi0FHZzHsO13CjhOowAa8mTUFTgWuBnD3EqAkqPVERA5k4/YShucW0KpJAx65LL3KOxIR5BZNZ6AYeNbMCs3sL2bWaM8HmdlQM8szs7zi4uIA44hIJisvd/4rXt7xxJXZNGuYXuUdiQhywNcFsoGn3L0vsA24bc8HufsYd89x95ysrKwA44hIJnvqg4W8O2ctd11wDH3aNws7TlIEOeBXACvcfUr89iRiA19EJKn+s3AdD705l+/1PoKrBqRneUciAhvw7v4lsNzMKt49cAYwK6j1RET2Zs3mndwwvpBOLRvxhx8el7blHYkI+iya64Hc+Bk0i4CfBryeiMjXKso7tu0qY9y1A9K6vCMRgf7XunsRkBPkGiIi+/Lgm/P4fPEGHr2sD91ap3d5RyKi995cERHgrVlr+NMHC7nyhA58v2/bsOOEQgNeRCJn2frt3DyxiJ5tm3LXBdEo70iEBryIRMrO0jKGj8sHolXekYjMesVBRCLvnkrlHe1bZPbVUXQELyKR8WLhCsZNWcYvvtMlcuUdidCAF5FImLdmC3dMnkH/Ti345dndwo6TEjTgRSTtbf26vKMujw/uS90IlnckQn8LIpLW3J3bJ09nybptPDa4L60iWt6RCA14EUlr/6hU3jGwy2Fhx0kpGvAikraKlm/kt6/M4vTurSJf3pEIDXgRSUtfbSthRLy84+FLe0e+vCMROg9eRNJOeblz08QiirfsYtKwgRlR3pEIHcGLSNp56oOFvD+3mLu+dwy92mVGeUciNOBFJK38Z0GsvOOiPkcw5IQOYcdJaRrwIpI21mzeyQ0TCumc1Zjf/yCzyjsSoT14EUkLpWXljBxXwLZdZYy/NptGGVbekYhA/4bMbAmwBSgDdru7yj9EJCEP/nsuXyz5itGX96FrBpZ3JCIZ3wIHufu6JKwjIhH15swv+d8PFzFkQAcu6pOZ5R2J0B68iKS0Zeu3c/PzU+nV7tCMLu9IRNAD3oE3zSzfzIbu7QFmNtTM8swsr7i4OOA4IpJOdpaWMSw3n4PMeOKKbA6um7nlHYkIesCf5O7ZwLnACDM7dc8HuPsYd89x95ysrKyA44hIOrn7X7OYuWozD1/aO+PLOxIR6IB391Xx39cCLwL9g1xPRKLjhfwVjP98GcNO68IZPVTekYjABryZNTKzJhUfA2cDM4JaT0SiY86Xm/n1S9M5oVMLbj5L5R2JCvIsmtbAi/E3ItQFxrn7GwGuJyIRsHXXbobnFtCkQT0eu0LlHTUR2IB390VA76CeX0Six9259YVpLFm3jXHXDqBVE5V31IS+NYpIyvj7p0t5ddpqfvXd7gzorPKOmtKAF5GUULjsK+59dRZn9mjFdad2DjtOJGjAi0joKso7WjdtwEOX9FF5Ry3R1XpEJFTl5c6NzxWxbmsJk4YN5NCG9cKOFBk6gheRUD3x3gI+mFfMf6u8o9ZpwItIaD5ZsI6H357H9/scwZUq76h1Vd6iMbODiJ32eASwA5jp7muCCiYi0fblpp2MmlDIUVmN+Z3KOwJxwAFvZl2AW4EzgflAMdAA6GZm24H/Bf7m7uVBBhWR6CgtK+f68QVsLyljwlCVdwSlKn+r9wJPAde5u1e+w8xaAVcAVwF/q/14IhJFD8TLO/44uC9HtVJ5R1AOOODdffB+7lsLPFqriUQk0t6Y8SVjPlzEVQOO5MLeR4QdJ9Kq/CKrmf3WzOpWut3UzJ4NJpaIRNHS9dv4Vby8484LeoQdJ/KqcxZNXWCKmfUys7OBL4D8YGKJSNTsLC1j2NgCDjpI5R3JUuVXNtz9djN7B5gCfAWc6u4LAksmIpFy979mMmv1Zp65OkflHUlSnS2aU4HRwD3A+8DjZqYNNBE5oEn5Kxj/+XJGDOrC6d1V3pEs1Tk36UHgEnefBWBmPwTeBboHEUxEomHOl5u586XpDOx8GDedqfKOZKrOgB/o7mUVN9x9spl9EEAmEYmILTtLGTY2Vt4xenAflXck2QH/ts1siJkdVHm4V3D39WbWxcxO3s/n1zGzQjN7paZhRSR9uDu3vTCdZRu28/jgvirvCEFVjuAPAwrNLJ/YWTMV72Q9CvgOsA64bT+fPwqYDTStWVQRSSd//c8SXp2+mtvO7c4JKu8IxQGP4N19NJANjAeygDPit1cCV7n7xe4+f2+fa2btgPOBv9RaYhFJeQXLvuL3r83mzB6tGXqKyjvCUqU9+Pj2zFvxX9XxKHALsM/3IpvZUGAoQIcOupqcSLrbsK2EkbkFtDm0AQ9d0lvlHSGqztUknwV8zz9392v28fgLgLXunm9mp+3red19DDAGICcn51vPLyLp4+vyjm0lTB52oso7Qlads2gqv0jaAPgBsGo/jz8JuNDMzos/vqmZjXX3IdWPKSLp4PH3FvDhvGJ+/4Pj6Nn20LDjZLzqvJP1hcq3zWw88PZ+Hn87cHv8sacBv9RwF4muj+ev45G35/GDvm0Z3L992HGEmjU6dQW0aS4irN60gxsmFNK1VWN+94OeKu9IEdXZg99CbA/e4r9/SawI5IDc/X1ilzcQkYgpLStn5LhCdpaW8eSV/WhYX+UdqaI6WzS6Kr+IfMv9b8whf2lFeUfjsONIJVWp7Mve3/3uXlB7cUQknbwxYzV//mgxPxmo8o5UVJUj+If2c58Dp9dSFhFJI0vWbeNXz0+jd/tm3HG+yjtSUVUq+wYlI4iIpI+dpWUMy60o7+ir8o4UVZ3rwTc0szvNbEz8dtf4m5lEJMP85uWZzF69mUcv60O75irvSFXVOU3yWaAEODF+ewVwb60nEpGU9nzecp7LW87IQUcxqHursOPIflRnwHdx9/uBUgB330HslEkRyRCzV2/mrpdncGKXw7jpLJV3pLrqDPgSMzuE+PVozKwLsCuQVCKScrbsLGV4bgFNG9Rj9OV9qaOLiKW8Kp0Hb7G3pf0JeANob2a5xK41c3Vw0UQkVbg7t74wjWUbtjP+2gFkNTk47EhSBVW9XLCb2SjgbGAAsa2ZUe6+LshwIpIanv1kCa9N/5I7zutO/04two4jVVSd9xR/BnR291eDCiMiqSd/aay846xjWnOtyjvSSnUG/CDgOjNbCmwjfk0ad+8VSDIRCd2GbSWMHFfA4c0a8OAlvXURsTRTnQF/bmApRCTllJU7oyYUsr6ivOMQlXekm+pcbGxpkEFEJLU8/u4CPpq/jj/8UOUd6aom14MXkYj6aH4xj74zjx9mt+Xy41Xeka404EXkG1Zv2sGoCUV0a9WEe7+v8o50FtiAN7MGZva5mU01s5lmdndQa4lI7SgtK2dEbgG7Sst4cki2yjvSXJD/eruA0919q5nVAz42s9fd/bMA1xSRGrjv9TkULNvIY4P70iVL5R3pLrAB7+4ObI3frBf/5UGtJyI18/r01Tz98WKuPrEj31N5RyQEugdvZnXMrAhYC7zl7lP28pihZpZnZnnFxcVBxhGRfVi8bhu3TJpGn/bNuOM8lXdERaAD3t3L3L0P0A7ob2Y99/KYMe6e4+45WVlZQcYRkb3YWVrGsLH51KljPHFlNvXr6tyLqEjKv6S7bwTeB85JxnoiUnX//fIM5q7ZwqOX9aFts0PCjiO1KMizaLLMrFn840OAM4E5Qa0nItU38YvlTMxbwfWDjuK0o1XeETVBnkVzOPA3M6tD7BvJRHd/JcD1RKQaZq2KlXecdNRhjDpT5R1RFORZNNOAvkE9v4gkbvPOUobn5tOsoco7okzvYhDJMO7OrZOmsfyrHUwYOoCWjVXeEVV6uVwkwzzzyRJen/Elt53TneM7qrwjyjTgRTJI/tIN/OG12Zx9TGt+fkqnsONIwDTgRTLE+q27GJFbSNvmh/CAyjsygvbgRTJAWblz43NFbNhewovDVd6RKXQEL5IB/vjOfD6av457LjyWY49QeUem0IAXibgP5xXzx3fnc3F2Oy5TeUdG0YAXibBVG3cwakIhR7dWeUcm0oAXiaiS3eWMHFdAaZnz5JXZHFK/TtiRJMn0IqtIRFWUdzxxRTadVd6RkXQELxJBr01fzTOfLOanJ3Xk/F6Hhx1HQqIBLxIxi4q3csukafTt0Izbz1V5RybTgBeJkB0lZQzPLaBeHePxK1Tekem0By8SIRXlHc9efbzKO0RH8CJRMfGL5Tyfv4LrT++q8g4BNOBFImHmqk3c9fIMTj6qJaPO6Bp2HEkRQVb2tTez98xstpnNNLNRQa0lksli5R0FNG9Yn9GX91F5h3wtyD343cDN7l5gZk2AfDN7y91nBbimSEZxd371/FRWxss7DlN5h1QS2BG8u69294L4x1uA2UDboNYTyURPf7yYf89cw23ndidH5R2yh6TswZtZR2L9rFP2ct9QM8szs7zi4uJkxBGJhLwlG7jv9Tl899jW/OxklXfItwU+4M2sMfACcKO7b97zfncf4+457p6TlZUVdByRSFi3dRcjx6m8Q/Yv0PPgzaweseGe6+6Tg1xLJFOUlTs3Tijiq+0lTB5+Ik0bqLxD9i6wAW+xQ4qngdnu/nBQ64hkmtHvzOfjBeu4/+JeKu+Q/Qpyi+Yk4CrgdDMriv86L8D1RCLv/blreezd+VzSrx2XqrxDDiCwI3h3/xjQxqBILVm1cQc3PVfE0a2bcM9FPcOOI2lA72QVSQMlu8sZofIOqSZdbEwkDfzh9dkULtvIk1eqvEOqTkfwIinu1WmrefaTJVxzUifOO07lHVJ1GvAiKWxh8VZumTSV7A7NuO3c7mHHkTSjAS+SonaUlDF8bAEH16uj8g5JiPbgRVKQu3PnSzOYt3YLf/tpf45QeYckQIcEIiloYt5yXihYwQ2nd+XUbrqEhyRGA14kxcTKO2ZySteW3KDyDqkBDXiRFLJpR6y8o0XD+jx6mco7pGa0By+SIiqXdzx3nco7pOZ0BC+SIv7y0WLenLWG28/rQb8jVd4hNacBL5ICvliygfvemMO5PdtwzUkdw44jEaEBLxKyWHlHAR1aNOT+H/VSeYfUGg14kRCVlTujJhSycXspT16ZTROVd0gt0ousIiEa/fY8Plmwnvt/1IsehzcNO45EjI7gRULy3ty1/PHdBVya045Lc1TeIbUvsAFvZs+Y2VozmxHUGiLpamW8vKN7G5V3SHCCPIL/K3BOgM8vkpZKdpczPLeAsjLnqSH9aFBP5R0SjMAGvLt/CGwI6vlF0tXvX5vN1OUbeeCSXnRq2SjsOBJhoe/Bm9lQM8szs7zi4uKw44gE6pVpq/jrf5bws5M7cU5PlXdIsEIf8O4+xt1z3D0nK0tXzZPoWli8lVsnTaPfkc1V3iFJEfqAF8kE20t2M2xsfry8oy/16uh/PQmezoMXCZi7c+eLM5i/dit/v6Y/hx+q8g5JjiBPkxwPfAocbWYrzOxnQa0lksomfLGcyYUrufGMbpzSVduQkjyBHcG7++CgnlskXcxYuYnf/HMmp3bL4vrTjwo7jmQYbQSKBKSivOOwRrHyjoNU3iFJpj14kQC4O798fiqrNu7guesG0qJR/bAjSQbSEbxIAP780SLemrWGO87rQb8jm4cdRzKUBrxILft88Qb+5425nHdcG36q8g4JkQa8SC0q3vJ/5R3/c7HKOyRcGvAitaSs3LlhfCGbd5by1BCVd0j49CKrSC155K15fLpoPQ9e0pvubVTeIeHTEbxILXhv7loef28Bl+W050f92oUdRwTQgBepsRVfbeem54rocXhT7r7o2LDjiHxNA16kBnbtLmPEuMJYeceV2SrvkJSiPXiRGvj9q7Hyjj8N6UdHlXdIitERvEiC/jl1FX/7dCnXntKJc3q2CTuOyLdowIskYMHardz2wjRyjmzOLeeovENSkwa8SDVtL9nN8Nx8DqlXh8evyFZ5h6Qs7cGLVIO78+t4ecc/rjmBNoc2CDuSyD7p0EOkGsZ/vpwXC1dy05ndOLlry7DjiOxXoAPezM4xs7lmtsDMbgtyLZGgTV+xif8XL+8YOUjlHZL6gqzsqwM8AZwLHAMMNrNjglpPJEibtpcyfFw+LRurvEPSR5B78P2BBe6+CMDMJgAXAbNqe6HvPfYxO0vLavtpRb62aUcpX20vUXmHpJUgB3xbYHml2yuAE/Z8kJkNBYYCdOjQIaGFumQ1oqSsPKHPFamqC3sfQXYHlXdI+ghywO/tZ1j/1h+4jwHGAOTk5Hzr/qp49PK+iXyaiEikBfki6wqgfaXb7YBVAa4nIiKVBDngvwC6mlknM6sPXA78M8D1RESkksC2aNx9t5mNBP4N1AGecfeZQa0nIiLfFOg7Wd39NeC1INcQEZG90ztZRUQiSgNeRCSiNOBFRCJKA15EJKLMPaH3FgXCzIqBpQl+ektgXS3GqS3KVX2pmk25qi9Vs0Up15HunrW3O1JqwNeEmeW5e07YOfakXNWXqtmUq/pSNVum5NIWjYhIRGnAi4hEVJQG/JiwA+yDclVfqmZTrupL1WwZkSsye/AiIvJNUTqCFxGRSjTgRUQiKlID3sz6mNlnZlZkZnlm1j/sTBXM7Pp4AflMM7s/7DyVmdkvzczNrGXYWQDM7AEzm2Nm08zsRTNrFnKelCyPN7P2Zvaemc2Of12NCjtTZWZWx8wKzeyVsLNUMLNmZjYp/vU128wGhp2pgpndFP93nGFm482sQU2fM1IDHrgfuNvd+wD/Hb8dOjMbRKyPtpe7Hws8GHKkr5lZe+AsYFnYWSp5C+jp7r2AecDtYQVJ8fL43cDN7t4DGACMSKFsAKOA2WGH2MNo4A137w70JkXymVlb4AYgx917ErvE+uU1fd6oDXgHmsY/PpTUaZAaBtzn7rsA3H1tyHkqewS4hb3UKYbF3d90993xm58RawMLy9fl8e5eAlSUx4fO3Ve7e0H84y3EhlXbcFPFmFk74HzgL2FnqWBmTYFTgacB3L3E3TeGm+ob6gKHmFldoCG1ML+iNuBvBB4ws+XEjpJDO/LbQzfgFDObYmYfmNnxYQcCMLMLgZXuPjXsLPtxDfB6iOvvrTw+JYZoZWbWEegLTAk3ydceJXbgUB52kEo6A8XAs/Gto7+YWaOwQwG4+0piM2sZsBrY5O5v1vR5Ay38CIKZvQ202ctdvwbOAG5y9xfM7FJi36nPTIFcdYHmxH6MPh6YaGadPQnnqB4g1x3A2UFn2Jv95XL3l+OP+TWxbYjcZGbbQ5XK48NkZo2BF4Ab3X1zCuS5AFjr7vlmdlrYeSqpC2QD17v7FDMbDdwG3BVuLDCz5sR+MuwEbASeN7Mh7j62Js+bdgPe3fc5sM3s78T2/QCeJ4k/Hh4g1zBgcnygf25m5cQuKlQcVi4zO47YF9NUM4PYNkiBmfV39y/DylUp30+AC4AzkvGNcD9SujzezOoRG+657j457DxxJwEXmtl5QAOgqZmNdfchIedaAaxw94qfciYRG/Cp4ExgsbsXA5jZZOBEoEYDPmpbNKuA78Q/Ph2YH2KWyl4ilgcz6wbUJ+Qr2bn7dHdv5e4d3b0jsS/+7GQM9wMxs3OAW4EL3X17yHFStjzeYt+ZnwZmu/vDYeep4O63u3u7+NfV5cC7KTDciX9tLzezo+N/dAYwK8RIlS0DBphZw/i/6xnUwgvAaXcEfwDXAqPjL1LsBIaGnKfCM8AzZjYDKAF+EvJRaap7HDgYeCv+08Vn7v6LMIKkeHn8ScBVwHQzK4r/2R3xLmTZu+uB3Pg360XAT0POA0B8y2gSUEBsW7KQWrhsgS5VICISUVHbohERkTgNeBGRiNKAFxGJKA14EZGI0oAXEYkoDXgRkYjSgBcRiSgNeJF9MLPj49ekb2BmjeLX6u4Zdi6RqtIbnUT2w8zuJXY9lUOIXcfkDyFHEqkyDXiR/Yi/pf0LYpe+ONHdy0KOJFJl2qIR2b8WQGOgCbEjeZG0oSN4kf0ws38Sa3HqBBzu7iNDjiRSZVG7mqRIrTGzHwO73X1cvJv1P2Z2uru/G3Y2karQEbyISERpD15EJKI04EVEIkoDXkQkojTgRUQiSgNeRCSiNOBFRCJKA15EJKL+P7+oay1v1UzwAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "x = torch.arange(-8.0, 8.0, 0.1, requires_grad=True)\n",
    "y = x.relu()\n",
    "xyplot(x, y, 'relu')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "graffitiCellId": "id_ltygoau",
    "id": "BFB05150DBD1474D9A9ECCB9CDF1DD39",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAX30lEQVR4nO3df7RdZX3n8fcniYiIgJKgTH6YsBqnRsYqc4lWW0BBF6gNXWtsSzq0trrM2CWWqjMWq6JiZ5xirWNHnGlUptUiFPFHM65Y6lR0pnVCk4CiCVKzqJIUHGJRarUKOeczf+x9w+nl/tg37H33fXI+r7WycvY5+577zcq993P383z388g2ERExvpb0XUBERPQrQRARMeYSBBERYy5BEBEx5hIEERFjblnfBczX8uXLvXbt2r7LiIgoyu7du79te8V0rxUXBGvXrmXXrl19lxERURRJ35zptQwNRUSMuQRBRMSYSxBERIy5BEFExJhLEEREjLnOgkDS1ZLulfTVGV6XpN+XtE/SbZLO6KqWiIiYWZdXBH8InD/L6xcA6+s/W4D/1mEtERExg87uI7D9vyWtneWUC4EPu1oHe4ekkySdavuermqKWAj3ff8BrtnxTR4cDPsuJY4y5z71ifzE6pNaf98+byhbCewfOT5QP/ewIJC0heqqgTVr1ixIcRFH6sY93+Ldn/0bAKSei4mjyiknHHvUBcF03yLT7pJjeyuwFWBiYiI76cSiNnklsPvN53Hy8Y/uuZqIufXZNXQAWD1yvAq4u6daIlozHFa/qyzJ5UAUos8g2Ab8ct099Gzg/swPxNFgUF+zJgiiFJ0NDUm6FjgHWC7pAPBW4FEAtv87sB14EbAP+AHwq13VErGQJvcBX5K7dKIQXXYNbZ7jdQOv7urzR/RlkKGhKEx+Z4loWZ0DLF2SIIgyJAgiWjash4ZyQRClSBBEtGyya2hpkiAKkSCIaNnAmSOIsiQIIlo2OUewJHMEUYgEQUTLhkOTDIiSJAgiWja00zEURUkQRLRsYKPMD0RBEgQRLbPTMRRlSRBEtGyQOYIoTIIgomVDOx1DUZQEQUTLqq6hBEGUI0EQ0bKhs85QlCVBENGygTNHEGVJEES0zM7QUJQlQRDRskHmCKIwCYKIlmWOIEqTIIho2XDo7EUQRUkQRLQsaw1FaRIEES0bOnsRRFkSBBEtqxad67uKiOYSBBEts51F56IoCYKIlqV9NEqTIIho2dDZpjLKkiCIaFm2qozSJAgiWpb20ShNgiCiZQOTrSqjKAmCiJZVXUN9VxHRXIIgomXpGorSJAgiWpatKqM0CYKIlg2HpGsoitJpEEg6X9IdkvZJumya19dIuknSrZJuk/SiLuuJWAjpGorSdBYEkpYCVwEXABuAzZI2TDntzcD1tp8JXAS8v6t6IhbKIDuURWG6vCLYCOyzfaftB4DrgAunnGPghPrxicDdHdYTsSCy+miUpssgWAnsHzk+UD836m3AxZIOANuB10z3RpK2SNoladfBgwe7qDWiNbmzOErTZRBM963gKcebgT+0vQp4EfARSQ+ryfZW2xO2J1asWNFBqRHtyRxBlKbLIDgArB45XsXDh35eAVwPYPv/AscCyzusKaJzg6FzZ3EUpcsg2Amsl7RO0jFUk8HbppxzF3AugKSnUgVBxn6iaDbZjyCK0lkQ2D4EXALcCNxO1R20R9IVkjbVp70eeKWkLwPXAr9ie+rwUURRBjZLcodOFGRZl29uezvVJPDoc5ePPN4LPLfLGiIW2jDto1GY/N4S0bJh1hqKwiQIIlo2NOkaiqIkCCJaVnUN9V1FRHMJgoiWOXMEUZgEQUTLBnbaR6MoCYKIlg1N2kejKPlyjWhZuoaiNAmCiJblPoIoTYIgomVpH43SJAgiWjZM+2gUJkEQ0bJhuoaiMAmCiJZVi84lCKIcCYKIlmWryihNgiCiZdmqMkqTIIhoWbaqjNIkCCJaZJuhyVaVUZQEQUSLJvfXS9dQlCRBENGiQZ0EGRmKkiQIIlo0nAyCJEEUpNGexZIeD/wL4J+Ab9gedlpVRKGG9XdG2kejJDMGgaQTgVcDm4FjgIPAscATJe0A3m/7pgWpMqIQk1cES3OtHQWZ7YrgBuDDwE/b/u7oC5L+NfBLkk6z/aEuC4woyUNzBLkiiHLMGAS2XzDLa7uB3Z1UFFEwZ2goCjTnBaykV0w5Xirprd2VFFGudA1FiZqMZJ4rabukUyWdDuwAHtdxXRFFemiOIEkQ5Ziza8j2L0r6BeArwA+Azbb/qvPKIgo0HFZBkDuLoyRNhobWA5cCHwe+QTVJfFzHdUUUqc6BzBFEUZoMDf1P4C22/x1wNvB1YGenVUUUapD20ShQkxvKNtr+BwDbBt4taVu3ZUWUKUNDUaIZf2+R9FMAkyEwyvbXJZ1QTx5HRO3wZHGCIAoy2wXsv5H0RUmXS3qxpI2SzpL0ckkfAT4NPGa2N5d0vqQ7JO2TdNkM5/y8pL2S9kj66CP4t0T07vAcQYaGoiCz3VD22nqNoZcCPwecSrXW0O3AH9j+y9neWNJS4CrgBcABYKekbbb3jpyzHngj8Fzb35F0yiP9B0X0aTDMncVRnlnnCGx/B/hA/We+NgL7bN8JIOk64EJg78g5rwSuqj8Ptu89gs8TsWg4S0xEgeacLJZ0+XTP275ijg9dCewfOT4APGvKOU+pP8dfAUuBt9n+s2lq2AJsAVizZs1cJUf0ZpAbyqJATUYyvz/yZwBcAKxt8HHTfSd4yvEyYD1wDtUqpx+UdNLDPsjeanvC9sSKFSsafOqIfjy0DHW/dUTMR5M7i989eizpd4Em7aMHgNUjx6uAu6c5Z4ftB4G/lXQHVTDkPoUo0jBDQ1GgI+ltOA44rcF5O4H1ktZJOga4iIcHyKeA5wFIWk41VHTnEdQUsSgkCKJETeYIvsJDQzpLgRXAXPMD2D4k6RLgxvrjrra9R9IVwC7b2+rXXihpL9Ww03+w/fdH9k+J6N9k+2jmCKIkTe4sfsnI40PA/7N9qMmb294ObJ/y3OUjjw28rv4TUbzB4TuLey4kYh5m26ryCfXD70156QRJ2L6vu7IiyuR0DUWBZrsi2E01JDRT90+TeYKIsZIbyqJEs91ZvG4hC4k4GmQZ6ihRk/0IJOliSW+pj9dI2th9aRHlGWaryihQk/bR9wM/Cfxiffw9qjWEImKKbFUZJWrSNfQs22dIuhWq9Yfq+wIiYopB9iOIAjW5IniwXknUAJJWAMNOq4oolHMfQRSoSRD8PvBJ4BRJ/xH4S+A/dVpVRKEe6hrquZCIeWiy1tA1knYD51K1kv6s7ds7ryyiQFliIko0axBIWgLcZvt04GsLU1JEuRIEUaJZh4ZsD4EvS8omABENZKvKKFGTrqFTgT2S/ppqTwIAbG/qrKqIQk3OEWTz+ihJkyB4e+dVRBwlJoeG0j4aJWkyWfyFhSgk4miQG8qiRBnJjGhRtqqMEiUIIlo0SNdQFGjGIJD0F/Xfv7Nw5USUbXI/giW5JIiCzDZHcKqks4FNkq5jyr4Etm/ptLKIAg3qoaF0DUVJZguCy4HLgFXA7015zcDzuyoqolRZhjpKNNvGNDcAN0h6i+13LGBNEcUaZmgoCtSkffQdkjYBZ9VPfd72p7stK6JMw2xVGQVqskPZO4FLgb31n0vr5yJiisHkMtQJgihIkzuLXww8o153CEl/BNwKvLHLwiJKNNk1pDRmR0GafrmeNPL4xC4KiTgaZK2hKFGTK4J3ArdKuomqhfQscjUQMa3Dq48mCKIgTSaLr5X0eeBMqiD4Tdvf6rqwiBI91DXUcyER89DkigDb9wDbOq4lonjpGooS5feWiBZNrjWUOYIoSYIgokWTcwTJgSjJjENDkp4w2wfavq/9ciLKZpslysY0UZbZ5gh2U60pJGAN8J368UnAXcC6zquLKMxg6MwPRHFmHBqyvc72acCNwM/YXm77ZOAlwCeavLmk8yXdIWmfpMtmOe+lkixpYr7/gIjFZOhMFEd5mswRnGl7++SB7c8AZ8/1QZKWAlcBFwAbgM2SNkxz3uOAXwdublp0xGI1tNM6GsVp8iX7bUlvlrRW0pMlvQn4+wYftxHYZ/tO2w8A1wEXTnPeO4ArgR82rjpikRpmaCgK1CQINgMrgE8CnwJOqZ+by0pg/8jxgfq5wyQ9E1g912qmkrZI2iVp18GDBxt86oh+DOy0jkZxmtxZfB/V6qPzNd13gw+/KC0B3gP8SoMatgJbASYmJjzH6RG9sdM6GuWZMwgkrQDeADwNOHbyedtz7VB2AFg9crwKuHvk+HHA6cDn61a7JwHbJG2yvatR9RGLzGBolmZTmihMk6Gha4CvUbWLvh34BrCzwcftBNZLWifpGOAiRpapsH1/3Ym01vZaYAeQEIiiDZ05gihPkyA42faHgAdtf8H2y4Fnz/VBtg8Bl1C1n94OXG97j6Qr6h3PIo46VddQgiDK0mTRuQfrv++R9GKq4Z1VTd68bjvdPuW5y2c495wm7xmxmA2H2bg+ytMkCH5b0onA64H/CpwAvLbTqiIKla6hKNGsQVDfFLa+bu+8H3jeglQVUaihnXWGojizzhHYHgAZz49oaJiuoShQk6GhL0p6H/AnwPcnn7R9S2dVRRSqWmuo7yoi5qdJEDyn/vuKkecMzHUfQcTYGaRrKArU5M7izAtENOTcRxAFanJn8eumefp+YLftL7VfUkS5BsN0DUV5mtxQNgG8imrBuJXAFuAc4AOS3tBdaRHlGWatoShQkzmCk4EzbP8jgKS3AjcAZ1HtYnZld+VFlCVdQ1GiJlcEa4AHRo4fBJ5s+5+AH3VSVUShstZQlKjJFcFHgR2S/rQ+/hngWkmPBfZ2VllEgQYmXUNRnCZdQ++QtB34Kao9Bl41skLov+2yuIjSVF1DfVcRMT9NrgiwvZtqPiAiZpGuoShRttmOaFHmCKJECYKIFg2HaR+N8iQIIlo0dNpHozwJgogWDTI0FAVKEES0aJj20ShQgiCiRWkfjRIlCCJalPbRKFGCIKJF1aJzCYIoS4IgokXVonN9VxExP/mSjWhRbiiLEiUIIlqUrSqjRAmCiBbZ5IogipMgiGhR1TXUdxUR85MgiGhR5giiRAmCiBYNh5kjiPIkCCJaNDS5sziKkyCIaNEgq49GgToNAknnS7pD0j5Jl03z+usk7ZV0m6S/kPTkLuuJ6Jrt3FkcxeksCCQtBa4CLgA2AJslbZhy2q3AhO2nAzcAV3ZVT8RCyFpDUaIurwg2Avts32n7AeA64MLRE2zfZPsH9eEOYFWH9UR0LnMEUaIug2AlsH/k+ED93ExeAXxmuhckbZG0S9KugwcPtlhiRLvSNRQl6jIIpvtu8LQnShcDE8C7pnvd9lbbE7YnVqxY0WKJEe3KfQRRomUdvvcBYPXI8Srg7qknSToPeBNwtu0fdVhPROfSNRQl6vKKYCewXtI6SccAFwHbRk+Q9EzgD4BNtu/tsJaIBVHtR9B3FRHz01kQ2D4EXALcCNwOXG97j6QrJG2qT3sXcDzwMUlfkrRthreLKMJwmKGhKE+XQ0PY3g5sn/Lc5SOPz+vy80cstKHTPhrlyZ3FES2xnfbRKFKCIKIlrnvi0j4apUkQRLRkUCdB5giiNAmCiJYM6yBI+2iUJkEQ0ZLhsPo7FwRRmgRBREsOXxEkCaIwCYKIlmSOIEqVIIhoieuhoXQNRWkSBBEtGR6+Iui5kIh5ShBEtGSQrqEoVIIgoiWTVwTZqjJKkyCIaMlk+2i6hqI0CYKIlmSOIEqVIIhoyWBYB0GSIAqTIIhoyeFF5zI0FIVJEES05KGuoZ4LiZinfMlGtGSYO4ujUAmCiJYMhwmCKFOCIKIlw8wRRKESBBEtmewayhxBlCZfshEtyZ3FUaoEQURLMlkcpUoQRLRkco4gQ0NRmnzJRrRkco4gQ0NRmgRBREucrSqjUAmCiJYMch9BFCpBENGSw/cR5LsqCpMv2YiWpGsoSpUgiGjJMFtVRqESBBEteWiOoOdCIuYpQRDRkuxHEKXqNAgknS/pDkn7JF02zeuPlvQn9es3S1rbZT0RXUrXUJSqsyCQtBS4CrgA2ABslrRhymmvAL5j+8eA9wC/01U9EV3LHEGUalmH770R2Gf7TgBJ1wEXAntHzrkQeFv9+AbgfZLkyTtzWnT9zv184P/c2fbbRhz2jz86BEAuCKI0XQbBSmD/yPEB4FkznWP7kKT7gZOBb4+eJGkLsAVgzZo1R1TMScc9ivVPPP6IPjaiqXMe8yh+7JR8nUVZugyC6X4vmvqbfpNzsL0V2AowMTFxRFcLL3zak3jh0550JB8aEXFU63Ky+ACweuR4FXD3TOdIWgacCNzXYU0RETFFl0GwE1gvaZ2kY4CLgG1TztkGvKx+/FLgc13MD0RExMw6Gxqqx/wvAW4ElgJX294j6Qpgl+1twIeAj0jaR3UlcFFX9URExPS6nCPA9nZg+5TnLh95/EPg57qsISIiZpc7iyMixlyCICJizCUIIiLGXIIgImLMqbRuTUkHgW8e4YcvZ8pdy4vIYq0tdc3PYq0LFm9tqWv+jqS2J9teMd0LxQXBIyFpl+2JvuuYzmKtLXXNz2KtCxZvbalr/tquLUNDERFjLkEQETHmxi0ItvZdwCwWa22pa34Wa12weGtLXfPXam1jNUcQEREPN25XBBERMUWCICJizI1dEEh6hqQdkr4kaZekjX3XNEnSayTdIWmPpCv7rmcqSf9ekiUt77sWAEnvkvQ1SbdJ+qSkk3qu5/z6/2+fpMv6rGWSpNWSbpJ0e/11dWnfNY2StFTSrZI+3XctoySdJOmG+uvrdkk/2XdNAJJeW/8/flXStZKObeN9xy4IgCuBt9t+BnB5fdw7Sc+j2sP56bafBvxuzyX9M5JWAy8A7uq7lhGfBU63/XTgb4A39lWIpKXAVcAFwAZgs6QNfdUz4hDwettPBZ4NvHqR1DXpUuD2vouYxnuBP7P948BPsAhqlLQS+HVgwvbpVMv7t7J0/zgGgYET6scn8vBd0/rya8B/tv0jANv39lzPVO8B3sA0W4n2xfaf2z5UH+6g2gWvLxuBfbbvtP0AcB1VsPfK9j22b6kff4/qB9rKfquqSFoFvBj4YN+1jJJ0AnAW1X4p2H7A9nf7reqwZcBj6h0dj6Oln1/jGAS/AbxL0n6q37p7+y1yiqcAPy3pZklfkHRm3wVNkrQJ+DvbX+67llm8HPhMj59/JbB/5PgAi+QH7iRJa4FnAjf3W8lh/4Xql4th34VMcRpwEPgf9bDVByU9tu+ibP8d1c+su4B7gPtt/3kb793pxjR9kfS/gOl2qn8TcC7wWtsfl/TzVKl/3iKoaxnweKrL9zOB6yWdtlBbd85R228BL1yIOqaarS7bf1qf8yaqIZBrFrK2KTTNc4vm6knS8cDHgd+w/Q+LoJ6XAPfa3i3pnL7rmWIZcAbwGts3S3ovcBnwlj6LkvR4qqvMdcB3gY9Jutj2Hz/S9z4qg8D2jD/YJX2YalwS4GMs4GXpHHX9GvCJ+gf/X0saUi0sdbDP2iT9K6ovvC9Lgmr45RZJG21/q6+6Rup7GfAS4Nye97s+AKweOV7FIhl2lPQoqhC4xvYn+q6n9lxgk6QXAccCJ0j6Y9sX91wXVP+XB2xPXjndQBUEfTsP+FvbBwEkfQJ4DvCIg2Ach4buBs6uHz8f+HqPtYz6FFU9SHoKcAyLYOVD21+xfYrttbbXUn2TnLEQITAXSecDvwlssv2DnsvZCayXtE7SMVSTeNt6rglV6f0h4Hbbv9d3PZNsv9H2qvpr6iLgc4skBKi/tvdL+pf1U+cCe3ssadJdwLMlHVf/v55LS5PYR+UVwRxeCby3nmz5IbCl53omXQ1cLemrwAPAy3r+DbcE7wMeDXy2vlrZYftVfRRi+5CkS4Abqbo5rra9p49apngu8EvAVyR9qX7ut+r9xGNmrwGuqUP9TuBXe66HepjqBuAWqqHQW2lpqYksMRERMebGcWgoIiJGJAgiIsZcgiAiYswlCCIixlyCICJizCUIIiLGXIIgImLMJQgiHiFJZ9Z7Ihwr6bH1evGn911XRFO5oSyiBZJ+m2rNnMdQrVPzzp5LimgsQRDRgnopgp1Uy5Y8x/ag55IiGsvQUEQ7ngAcDzyO6sogohi5IohogaRtVLuSrQNOtX1JzyVFNDaOq49GtErSLwOHbH+03rv4i5Keb/tzfdcW0USuCCIixlzmCCIixlyCICJizCUIIiLGXIIgImLMJQgiIsZcgiAiYswlCCIixtz/B0lhayVd94bJAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "y.sum().backward()\n",
    "xyplot(x, x.grad, 'grad of relu')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_i6a7r84",
    "id": "4C49EE87FAB54B0B8F89B617C1689C69",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "#### Sigmoid函数\n",
    "sigmoid函数可以将元素的值变换到0和1之间：\n",
    "\n",
    "\n",
    "$$\n",
    "\\text{sigmoid}(x) = \\frac{1}{1 + \\exp(-x)}.\n",
    "$$\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "graffitiCellId": "id_a5oy7ht",
    "id": "68FCB4E8142144458F13128B370D1C91",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXxU9b3/8ddnsickIISdQEBZBBS1YXFrVdzX2t4qbnW70t5brVpvf7XtrW29v/7aa2t3u3iVat0Al1rai2vR1g0ICC5sEtkSdggQIHvm8/tjBjrGQAbM5Mxk3s/HYx4z58yZ5M2DybznbN9j7o6IiKSvUNABREQkWCoCEZE0pyIQEUlzKgIRkTSnIhARSXOZQQc4VMXFxV5aWhp0DBGRlLJw4cJt7t67redSrghKS0tZsGBB0DFERFKKma090HPaNCQikuZUBCIiaU5FICKS5lQEIiJpTkUgIpLmElYEZjbNzLaY2fsHeN7M7JdmVmFm75rZCYnKIiIiB5bINYKHgHMP8vx5wPDobSrw2wRmERGRA0jYeQTu/g8zKz3IIpcAf/TIONhzzayHmfV3942JyiQicjDuTlOL09gSpqGphcaWMI3N0VtLmJawf/zmTnPYCYcj9+0t4+6EHTz6+9wh7I4TvffY+eBElsedyUf3ZVxJjw7/dwd5QtlAoDJmuio672NFYGZTiaw1MHjw4E4JJyKpoa6xhR21jVTvbWRnbRPVtY3U1DWxt6GZvY0t7G1opraxmT0NkceR+c3UNrTQ0BymoTlyv+/DPlkv0WIGfbvndrkisDbmtflf4O73A/cDlJWVJel/k4h0pPqmFiqra9mwq55Nu+rYtKuBTTV1bNpVz+aaBnbUNrKjtpH6pvBBf05Bdgb5OZl0y8mkICeD/OxM+hTmktcrg9zMDHKyQmRnhMjJjNyyo7eczIzI44wQOVkhMkMhMkNGRoaRYUZmyAiFWt2bkZkR8zgUIhRi/32GReabgUXvQ2YYxMwHwwhFl9l3n0hBFkEVUBIzPQjYEFAWEQnIlpp6lmyooWLLHlZv38uabZHbxpr6j307L+6WTb/uufTvnsvoAUX0LMimR34WPfOz6ZGfTc+CbI7Iz6J7XhYFOZnkZWUQCiX2Q7QrCLIIZgE3m9l0YCKwS/sHRLq2DTvrWLRuJ0s27GLJhhqWbKhh256G/c/3yM+itFcBE4f1orRXAaXF+QzskUffolz6FuWSnakj3hMhYUVgZk8ApwHFZlYFfBfIAnD33wGzgfOBCqAWuD5RWUQkGBt31fHaB9uYt7qaeau3U7WjDoDMkDG8byGnjezNmAFFjBnQnRF9u9EjPzvgxOkpkUcNXdHO8w58JVG/X0Q6XzjsLK7ayZxlW/jb8i0s21gDQM+CbCaU9uSGk4dSVnoEI/sVkpOZEXBa2SflhqEWkeSzYtNunl28nlmLN7B+Zx0ZIeNTQ47gzvNGcdrI3ozsW5jwHZ5y+FQEInJY6hpbeHbxeh55ay1LN9aQETJOOaqYO84eweRRfemenxV0RImTikBEDklldS2PzF3LjPJKdtU1MapfId+7aDQXHDuA3oU5QceTw6AiEJG4rN9Zx6/nrOTJBVU4cO6Yflx7UinjS4/QZp8UpyIQkYPaUlPPr1+pYPr8yEAAV04czJc/cyQDeuQFnEw6iopARNrU3BLm4bfW8rOXPqC+qYUvlA3i5jOGM1AF0OWoCETkYxaurebbf3qf5Zt285kRvfn+xWMoLS4IOpYkiIpARParb2rhR88t56E319C/ey6/u/oEzhnTT/sAujgVgYgAsHLzbm55YhHLN+3mupNK+fo5IynI0UdEOtD/skiac3eml1fy/b8soSA7kz9cP57TR/YJOpZ0IhWBSBprbA7z7T+9x5MLqzh1eDH3XjaOPoW5QceSTqYiEElTu2qb+PKjC3lr1Xa+Onk4t00eriGb05SKQCQNrdtey/UPzWdddS0/vWwcnzthUNCRJEAqApE08/76XXxx2nzC7jx640QmDusVdCQJmIpAJI0s2bCLqx+cR0F2Jo/cOIFhvbsFHUmSgIpAJE0s3VDDVQ/MIz8rgydumsTgXvlBR5Ikoeu+iaSBSAnMJS8rgyemqgTko1QEIl3cmm17ufrBeeRmZTB96iSG9NJQEfJRKgKRLmxnbSM3PFSOu/P4TSoBaZv2EYh0UY3NYb786EKqdtTx6L9OZKgGjZMDUBGIdEHuzjefeY+5q6r52eXjmDC0Z9CRJIlp05BIF/SbVz/k6beruHXycC49XieLycGpCES6mDcqtvGTF1dw8bgB3Hbm8KDjSApQEYh0IVt3N3DbjMUMKy7gR58/RtcRkLhoH4FIFxEOO1+buZiauib+eMME8rP15y3x0RqBSBfxu398yGsrt/Hdi8ZwdP+ioONIClERiHQBC9dWc++LH3DBsf25YkJJ0HEkxagIRFJcXWMLt894h4E98vjh57RfQA6dNiKKpLh7X1zBuupapk+dRFFuVtBxJAVpjUAkhS1at4Npb6zmqomDmaTrCshhUhGIpKjG5jDfePpd+hblcud5o4KOIyksoUVgZuea2QozqzCzO9t4frCZvWJmi8zsXTM7P5F5RLqS37xawQeb9/CDS8dSqE1C8gkkrAjMLAO4DzgPGA1cYWajWy32n8BMdz8emAL8JlF5RLqSFZt2c98rFXz2uAGcMapv0HEkxSVyjWACUOHuq9y9EZgOXNJqGQf2HfDcHdiQwDwiXYK7850/v0+3nEzuumhM0HGkC0hkEQwEKmOmq6LzYn0PuNrMqoDZwC1t/SAzm2pmC8xswdatWxORVSRlPPf+JuavruaOs0fSsyA76DjSBSSyCNo6mNlbTV8BPOTug4DzgUfM7GOZ3P1+dy9z97LevXsnIKpIaqhvauH/zV7GqH6FTBmvE8ekYySyCKqA2HfqID6+6edGYCaAu78F5ALFCcwkktIefH01VTvquOvC0WRm6KA/6RiJfCeVA8PNbKiZZRPZGTyr1TLrgMkAZnY0kSLQth+RNmyuqee+Vyo4Z0xfTjpK35ek4ySsCNy9GbgZeAFYRuTooCVmdreZXRxd7A7gJjN7B3gCuM7dW28+EhHgnudX0NzifPv81gffiXwyCR1iwt1nE9kJHDvvrpjHS4GTE5lBpCt4r2oXT79dxZc/cySDe+UHHUe6GG1kFEkB97ywnJ4F2Xzl9CODjiJdkIpAJMnNW7Wd11Zu498+c6TOIJaEUBGIJDF3594XP6BPYQ5XTxoSdBzpolQEIkns9YptzF9Tzc1nHEVedkbQcaSLUhGIJCl35ycvrGBgjzwu18ljkkAqApEk9fKyLbxTtYtbJw8nJ1NrA5I4KgKRJBQOO/e+uIKhxQV87oTWQ3SJdCwVgUgSenHpZpZv2s1tZw7XUBKScHqHiSQZd+e3r1YwpFc+FxzTP+g4kgZUBCJJ5s0Pt/NO1S6+9OkjtTYgnULvMpEk89tXP6R3YY72DUinURGIJJF3q3byesU2/vWUoeRm6Ugh6RwqApEk8ptXPqQoN5MrJw4OOoqkERWBSJKo2LKHF5Zu4osnlmpMIelUKgKRJPH7v39IdkaI604uDTqKpBkVgUgS2FxTz7OL13P5+BKKu+UEHUfSjIpAJAk8OnctzWHnxlOGBh1F0pCKQCRg9U0tPDZvHZNH9WVIr4Kg40gaUhGIBOzPi9dTvbeRG04pDTqKpCkVgUiA3J1pr69hVL9CThzWK+g4kqZUBCIBeuvD7azYvJsbThmKmQUdR9KUikAkQNPeWE2vgmwuHjcg6CiSxlQEIgFZvW0vf1u+hasmDtZwEhIoFYFIQB5+cw2ZIdNF6SVwKgKRAOxpaOaphVVceOwA+hTlBh1H0pyKQCQAzy5az56GZq45UWsDEjwVgUgnc3cenbuW0f2LOL6kR9BxRFQEIp1t4dodLN+0m2tOHKJDRiUpqAhEOtmjc9dSmJPJJcfpkFFJDpnxLmhmRwADgDpgjbuHE5ZKpIvavqeB2e9t4sqJg8nPjvvPTyShDrpGYGbdzexbZvYeMBf4PTATWGtmT5rZ6e28/lwzW2FmFWZ25wGWuczMlprZEjN7/HD/ISKpYOaCKhpbwlylK5BJEmnvK8lTwB+BU919Z+wTZvYp4BozG+buD7Z+oZllAPcBZwFVQLmZzXL3pTHLDAe+CZzs7jvMrM8n++eIJK+WsPP4/LVMGtaT4X0Lg44jst9Bi8DdzzrIcwuBhQd5+QSgwt1XAZjZdOASYGnMMjcB97n7jujP3BJnbpGU848PtlJZXcc3zh0VdBSRj4hrZ7GZ3dhqOsPMvtvOywYClTHTVdF5sUYAI8zsDTOba2bnHuD3TzWzBWa2YOvWrfFEFkk6j81bS3G3HM4e3S/oKCIfEe9RQ5PNbLaZ9TezsUT2F7S3btvWcXHeajoTGA6cBlwBPGBmHzuw2t3vd/cydy/r3bt3nJFFksemXfXMWb6Fy8oGkZ2pg/UkucR12IK7X2lmlwPvAbXAFe7+RjsvqwJKYqYHARvaWGauuzcBq81sBZFiKI8nl0iqePrtKsIOl5WVtL+wSCeLd9PQcOBW4GlgDZGdxPntvKwcGG5mQ80sG5gCzGq1zLPA6dHfUUxkU9GquNOLpIBw2JlRXsmJw3pRWqxLUUryiXcd9S/Ad9z9S8BngJW0863d3ZuBm4EXgGXATHdfYmZ3m9nF0cVeALab2VLgFeDr7r79MP4dIklr7qrtrKuuZcoErQ1Icor3jJYJ7l4D4O4O3Gtmrb/df4y7zwZmt5p3V8xjB74WvYl0SdPLK+mel8U5Y7STWJJTeyeUnQKwrwRiuftKMyuK7jwWkTbs2NvI8+9v4tLjB+riM5K02lsj+LyZ3QM8T+Scga1ALnAUkW37Q4A7EppQJIU9u3g9jS1hLh+vzUKSvNo7oez26BhD/wJ8AehPZKyhZcDv3f31xEcUSU3uzvT5lYwr6cHR/YuCjiNyQO3uI4ie9fs/0ZuIxGlx5U5WbN7NDz93TNBRRA7qoEVgZgfdievuP+3YOCJdx4zySvKzM7honIabluTW3hrBvrOHRwLj+ed5ABcB/0hUKJFUt6ehmVnvbODCY/vTLUfDTUtya28fwfcBzOxF4AR33x2d/h7wZMLTiaSo/313A7WNLVw+XsNNS/KL94SywUBjzHQjUNrhaUS6iOnllQzv040TBuuaxJL84l1nfQSYb2Z/IjJw3KVErlMgIq2s2LSbRet28p8XHK1rEktKiHfQuR+Y2XPAqdFZ17v7osTFEkldM8orycowPnfCoKCjiMSlvaOGity9xsx6Ehlsbk3Mcz3dvTqx8URSS0NzC88squLsMf3oWZAddByRuLS3RvA4cCGRs4qdj15jwIFhCcolkpJeXLKZnbVNTNGZxJJC2jtq6MLo/dDOiSOS2maUVzKwRx4nH1kcdBSRuMV9gHN06OhPRydfdfe/JiaSSGpat72W1yu28bWzRhAKaSexpI54L0zzIyIXplkavd1qZj9MZDCRVDNzQSUhgy+UaSexpJZ41wjOB45z9zCAmT0MLAK+mahgIqmkuSXMkwsrOW1kH/p3zws6jsghOZSraMeeGdO9o4OIpLK/f7CVzTUNGm5aUlK8awQ/BBaZ2StEjhz6NFobENlvenklxd1yOGNUn6CjiByyeE8oe8LMXiUy8JwB33D3TYkMJpIqttTUM2f5Fm46dRhZGYeyki2SHA7lXds7ep8BnGRmn0tAHpGU89TbVbSEXZuFJGXFtUZgZtOAY4ElQDg624FnEpRLJCW4OzPKK5k4tCdDiwuCjiNyWOLdRzDJ3UcnNIlICpq7qpq122u57czhQUcROWzxbhp6y8xUBCKtzChfR2FuJueN7R90FJHDFu8awcNEymAT0EBkh7G7+7EJSyaS5HbWNjL7/U1MGV9CblZG0HFEDlu8RTANuAZ4j3/uIxBJa8+8vZ7G5rB2EkvKi7cI1rn7rPYXE0kP7s4T89cxrqQHYwbo/EpJbfEWwXIzexz4C5FNQwC4u44akrS0cO0OVm7Zwz2f19ZRSX3xFkEekQI4O2aeDh+VtPX4vHV0y8nkwnHaSSypL94zi69PdBCRVLGztpG/vreRy8oGkZ8d90juIkkr3hPKftnG7F3AAnf/c8dGEkluf1oU2Ul85YQhQUcR6RDxnkeQCxwHrIzejgV6Ajea2c8TlE0k6bg7j8+L7CQePaAo6DgiHSLeIjgKOMPdf+XuvwLOBI4GLuWj+w0+wszONbMVZlZhZnceZLl/MTM3s7JDCS/S2fbtJL5ygg4Zla4j3iIYCMQOpFIADHD3FmKOIoplZhnAfcB5wGjgirbOTjazQuCrwLxDyC0SiMfnR3YSXzRuQNBRRDpMvEVwD7DYzP5gZg8RuTrZT8ysAHj5AK+ZAFS4+yp3bwSmA5e0sdx/RX9+/SElF+lku2qb+N93N/LZ4wdoJ7F0KXEVgbs/CJwEPBu9neLuD7j7Xnf/+gFeNhCojJmuis7bz8yOB0rc/a8H+/1mNtXMFpjZgq1bt8YTWaTDPbOoiobmMFdMGBx0FJEOddAiMLNR0fsTgP5EPtjXAf2i8w768jbmeczPDgE/A+5oL6S73+/uZe5e1rt37/YWF+lwOpNYurL21m+/BkwF7o2Z5zGPzzjIa6uA2D1qg4ANMdOFwFjgVTMD6AfMMrOL3X1BO7lEOtXCtTv4YPMe/vvzxwQdRaTDHXSNwN2nRh/+FrjE3U8HXiFyDsF/tPOzy4HhZjbUzLKBKcD+8YrcfZe7F7t7qbuXAnMBlYAkpX07iS88VjuJpeuJd2fxf7p7jZmdApwFPESkHA7I3ZuBm4EXgGXATHdfYmZ3m9nFnyCzSKfasbdx/07ighztJJauJ953dUv0/gLgd+7+ZzP7XnsvcvfZwOxW8+46wLKnxZlFpFNNL6+koTnMNZNKg44ikhDxrhGsN7PfA5cBs80s5xBeK5KyWsLOo3PXcuKwXozsVxh0HJGEiPfD/DIim3jOdfedRIaXONBhoyJdxt+WbWb9zjquPUnjCknXFe/oo7XEDDnt7huBjYkKJZIsHn5rDQO653Lm0X2DjiKSMNq8I3IAFVt280bFdq6aNITMDP2pSNeld7fIATz85lqyM0NM0TWJpYtTEYi0oaa+iaffruKiYwfQq1tO0HFEEkpFINKGZxZWUdvYop3EkhZUBCKttISdh95cw3ElPTh2UI+g44gknIpApJWXlm5mzfZabjp1WNBRRDqFikCklf95bRUlPfM4Z4wOGZX0oCIQibFw7Q4Wrt3BjScP1SGjkjb0TheJ8cBrq+iel8UXynTIqKQPFYFI1Nrte3l+ySaumjhYo4xKWlERiEQ9+PpqMkPGdSeVBh1FpFOpCESIXHNg5oJKPnvcQPoU5QYdR6RTqQhEgEfmrqW+KcxNn9Yho5J+VASS9vY0NPPg66uZPKoPI/rqmgOSflQEkvb++NYadtU18dXJw4OOIhIIFYGktb0NzTzw2mpOG9mbcSUaTkLSk4pA0tpj89ZSvbeRW87Q2oCkLxWBpK26xhbu/8cqTh1ezKeGHBF0HJHAqAgkbT0+fx3b9jRq34CkPRWBpKX6phZ+9/cPOXFYL8aX9gw6jkigVASSlh6bt46tuxu4ZfJRQUcRCZyKQNJOTX0Tv56zklOOKuakI4uDjiMSOBWBpJ3fvfohO2qbuPO8UUFHEUkKKgJJKxt31fHg66v57HEDGDuwe9BxRJKCikDSys9e+gB3uOPskUFHEUkaKgJJGx9s3s1TC6v44olDKOmZH3QckaShIpC08d/PLacgJ5OvnK4jhURiJbQIzOxcM1thZhVmdmcbz3/NzJaa2btm9jczG5LIPJK+Xl+5jb8t38K/n3YURxRkBx1HJKkkrAjMLAO4DzgPGA1cYWajWy22CChz92OBp4B7EpVH0ldDcwt3/fl9Snvlc/3JpUHHEUk6iVwjmABUuPsqd28EpgOXxC7g7q+4e210ci4wKIF5JE39zz9WsWrbXu6+ZCy5WRlBxxFJOoksgoFAZcx0VXTegdwIPNfWE2Y21cwWmNmCrVu3dmBE6eoqq2v51ZwKLjimP58e0TvoOCJJKZFFYG3M8zYXNLsaKAN+3Nbz7n6/u5e5e1nv3vpjlvi4O9+dtYTMkPGdC1tvlRSRfRJZBFVAScz0IGBD64XM7Ezg28DF7t6QwDySZl5aupk5y7dw+1kj6NddF6QXOZBEFkE5MNzMhppZNjAFmBW7gJkdD/yeSAlsSWAWSTO765v4/l+WMqpfIdeeVBp0HJGklrAicPdm4GbgBWAZMNPdl5jZ3WZ2cXSxHwPdgCfNbLGZzTrAjxM5JP/116Vs3FXHDy49hqwMnS4jcjCZifzh7j4bmN1q3l0xj89M5O+X9PTikk3MXFDFV04/UlceE4mDvipJl7JtTwPffOY9xgwo4tbJI4KOI5ISErpGINKZ3J07n36P3Q3NPHH5cWRn6nuOSDz0lyJdxpMLqnh52Wb+zzkjGdG3MOg4IilDRSBdwvJNNXx31hImDevJDScPDTqOSEpREUjK21XbxJceWUhhbia/nHI8oVBb5zKKyIFoH4GktJawc+uMRWzYWcf0qZPoU6QTx0QOldYIJKX9/OUPeHXFVr570Rg+NaRn0HFEUpKKQFLW8+9v4ldzKrisbBBXTRwcdByRlKUikJRUvqaaW6cvYlxJD+6+ZCxm2i8gcrhUBJJylm2s4YaHyhnYI49p15bpGgMin5CKQFLK2u17+eK0+XTLyeSRf51Ir245QUcSSXkqAkkZW2rquebB+TS1hHnkxgkM7JEXdCSRLkFFICmhsrqWy37/Ftv2NPCH68ZzVB+dOSzSUXQegSS9lZt3c/WD86hrbOGRGydy/GCNKCrSkVQEktQWV+7kuj/MJysjxMwvn8iofkVBRxLpclQEkrTmLN/MLY8vome3bB69cSJDehUEHUmkS1IRSNIJh51fzlnJz19eyZgBRUy7bjx9NXSESMKoCCSp7Kpr4vYZi5mzfAufP2EQP7h0rM4TEEkwFYEkjbfX7eD2GYtZv6OO/7pkDFdPGqIzhkU6gYpAAlff1MJPX/qAB15bRf/ueUyfOomyUg0gJ9JZVAQSqIVrq/n6U++yauterpw4mG+dfzTdcvS2FOlM+ouTQKzfWcePn1/Os4s3MLBHHo/eOJFThhcHHUskLakIpFPtaWjmt69W8MBrq3Hg3087kn8//SitBYgESH990imq9zby8JtrePitNeysbeKzxw3g6+eO0nhBIklARSAJVVldy4Ovr2ZGeSV1TS2ceXRfbjnjKMaV9Ag6mohEqQikw9U3tfDi0s3MKF/HGxXbyQwZnz1+IF/69DCG99VgcSLJRkUgHaKpJcybH27n+fc38tz7m9hZ28TAHnl87awRfKFsEP27axOQSLJSEchh27angTc/3M6rK7bw8tLN1NQ3U5CdweSj+3JZWQknHdmLUEgnhIkkOxWBxG1LTT1vr9vJwrXVvFGxnaUbawDonpfFWaP7cd7YfpwyvFhDQoikGBWBfEw47FTuqGXFpt2s2LSb5Zt3s3jdTtbvrAMgOyPECUN68PVzRnLKUcWMHdidDH3zF0lZKoI0FQ47m3fXU1ldR2V1Leuqa6ncUcuHW/eycvNuahtb9i9b0jOP40p6cP3JpRw/+AjGDiwiJ1Pf+kW6ioQWgZmdC/wCyAAecPcftXo+B/gj8ClgO3C5u69JZKauyt1paA5TU9dETX0Tu+qa2Lq7kW17Gti6u4Gt++5jbo0t4f2vN4P+RbmUFhdw+fgSRvYtZGS/Qkb0LaRAJ3uJdGkJ+ws3swzgPuAsoAooN7NZ7r40ZrEbgR3ufpSZTQH+G7g8UZkSxd0JO4Td8eh9S9hpagnT1BK5b25xmsLh/Y8b981rCe9frrklTGP0cV1TC3WNzdQ1hv/5uKmFuqbw/sd7GlrYHf3gr6lr/sgHeywz6JmfTe/CHHoX5jCsuIDeRTmUHJFPSc98BvfMZ0CPXH3LF0lTifyqNwGocPdVAGY2HbgEiC2CS4DvRR8/BfzazMzdvaPDzCyv5P7XVn3kwzrsTjj80Q/ysO+bdpzIJhSPee5jr+/wpB+XnREiLzuDvKyMj9wX5WZSckQeRXlZFOVmUZSXGb3Poig3k+JuOfQpzKFnQTaZGaHEBxWRlJTIIhgIVMZMVwETD7SMuzeb2S6gF7AtdiEzmwpMBRg8ePBhhTmiIJuRfQsxg5AZoei9xTwOhdg/bUTvzf65fMgO+nrjn8tkmJGVESIrI3KfGfM4Mm1kZ4TIDBlZmSGyQiGyMo3MUIjsjBC52SHyszPJzQzpQ1xEEiqRRdDWYSStvz/Hswzufj9wP0BZWdlhfQc/a3Rfzhrd93BeKiLSpSXyq2YVUBIzPQjYcKBlzCwT6A5UJzCTiIi0ksgiKAeGm9lQM8sGpgCzWi0zC7g2+vhfgDmJ2D8gIiIHlrBNQ9Ft/jcDLxA5fHSauy8xs7uBBe4+C3gQeMTMKoisCUxJVB4REWlbQg8Qd/fZwOxW8+6KeVwPfCGRGURE5OB0OIqISJpTEYiIpDkVgYhImlMRiIikOUu1ozXNbCuw9jBfXkyrs5aTSLJmU65Dk6y5IHmzKdehO5xsQ9y9d1tPpFwRfBJmtsDdy4LO0ZZkzaZchyZZc0HyZlOuQ9fR2bRpSEQkzakIRETSXLoVwf1BBziIZM2mXIcmWXNB8mZTrkPXodnSah+BiIh8XLqtEYiISCsqAhGRNJd2RWBmx5nZXDNbbGYLzGxC0Jn2MbNbzGyFmS0xs3uCztOamf2HmbmZFQedBcDMfmxmy83sXTP7k5n1CDjPudH/vwozuzPILPuYWYmZvWJmy6Lvq1uDzhTLzDLMbJGZ/TXoLLHMrIeZPRV9fy0zsxODzgRgZrdH/x/fN7MnzCy3I35u2hUBcA/wfXc/DrgrOh04MzudyDWcj3X3McBPAo70EWZWApwFrAs6S4yXgLHufizwAfDNoIKYWQZwH3AeMBq4wsxGB5UnRjNwh7sfDUwCvpIkufa5FVgWdIg2/AJ43t1HAeNIgoxmNhD4KlDm7mOJDO/fIUP3p+9qscoAAAPLSURBVGMROFAUfdydj181LSj/BvzI3RsA3H1LwHla+xnwf2jjUqJBcfcX3b05OjmXyFXwgjIBqHD3Ve7eCEwnUuyBcveN7v529PFuIh9oA4NNFWFmg4ALgAeCzhLLzIqATxO5Xgru3ujuO4NNtV8mkBe9omM+HfT5lY5FcBvwYzOrJPKtO7Bvka2MAE41s3lm9nczGx90oH3M7GJgvbu/E3SWg7gBeC7A3z8QqIyZriJJPnD3MbNS4HhgXrBJ9vs5kS8X4aCDtDIM2Ar8IbrZ6gEzKwg6lLuvJ/KZtQ7YCOxy9xc74mcn9MI0QTGzl4F+bTz1bWAycLu7P21mlxFp/TOTIFcmcASR1ffxwEwzG9ZZl+5sJ9u3gLM7I0drB8vl7n+OLvNtIptAHuvMbK1YG/OSZu3JzLoBTwO3uXtNEuS5ENji7gvN7LSg87SSCZwA3OLu88zsF8CdwHeCDGVmRxBZyxwK7ASeNLOr3f3RT/qzu2QRuPsBP9jN7I9EtksCPEknrpa2k+vfgGeiH/zzzSxMZGCprUFmM7NjiLzx3jEziGx+edvMJrj7pqByxeS7FrgQmBzw9a6rgJKY6UEkyWZHM8siUgKPufszQeeJOhm42MzOB3KBIjN71N2vDjgXRP4vq9x935rTU0SKIGhnAqvdfSuAmT0DnAR84iJIx01DG4DPRB+fAawMMEusZ4nkwcxGANkkwciH7v6eu/dx91J3LyXyR3JCZ5RAe8zsXOAbwMXuXhtwnHJguJkNNbNsIjvxZgWcCYu094PAMnf/adB59nH3b7r7oOh7agowJ0lKgOh7u9LMRkZnTQaWBhhpn3XAJDPLj/6/TqaDdmJ3yTWCdtwE/CK6s6UemBpwnn2mAdPM7H2gEbg24G+4qeDXQA7wUnRtZa67fzmIIO7ebGY3Ay8QOZpjmrsvCSJLKycD1wDvmdni6LxvRa8nLgd2C/BYtNRXAdcHnIfoZqqngLeJbApdRAcNNaEhJkRE0lw6bhoSEZEYKgIRkTSnIhARSXMqAhGRNKciEBFJcyoCEZE0pyIQEUlzKgKRT8jMxkeviZBrZgXR8eLHBp1LJF46oUykA5jZ/yUyZk4ekXFqfhhwJJG4qQhEOkB0KIJyIsOWnOTuLQFHEombNg2JdIyeQDegkMiagUjK0BqBSAcws1lErko2FOjv7jcHHEkkbuk4+qhIhzKzLwLN7v549NrFb5rZGe4+J+hsIvHQGoGISJrTPgIRkTSnIhARSXMqAhGRNKciEBFJcyoCEZE0pyIQEUlzKgIRkTT3/wFbqM3W/yGNXAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "y = x.sigmoid()\n",
    "xyplot(x, y, 'sigmoid')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_cf0vxx0",
    "id": "752D25727EC14654884CB9CB6515FC87",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "依据链式法则，sigmoid函数的导数\n",
    "\n",
    "\n",
    "$$\n",
    "\\text{sigmoid}'(x) = \\text{sigmoid}(x)\\left(1-\\text{sigmoid}(x)\\right).\n",
    "$$\n",
    "\n",
    "\n",
    "下面绘制了sigmoid函数的导数。当输入为0时，sigmoid函数的导数达到最大值0.25；当输入越偏离0时，sigmoid函数的导数越接近0。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "graffitiCellId": "id_1ps84t4",
    "id": "878C7B8823304F72860965E119A21412",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3dd3xc1Zn/8c8jjXqxVV1kyZYsG1um2ZYrLphqmknYEEpIgJCwZEOSTbKbH1mS8Fuyv5CEDZvsht2EDeyGhBpIwJTQi8ENyxV3SzK2ZdmyilVs1Zl5fn/MyAxiJI0sje5Iet6v17w0c+femW9ioefec+45R1QVY4wxpqsopwMYY4yJTFYgjDHGBGUFwhhjTFBWIIwxxgRlBcIYY0xQLqcDDJTMzEydNGmS0zGMMWZI2bhxY42qZgV7b9gUiEmTJlFSUuJ0DGOMGVJE5EB371kTkzHGmKCsQBhjjAnKCoQxxpigrEAYY4wJygqEMcaYoMJaIERkuYjsEZFSEbkryPvfEZGdIrJNRN4UkYkB73lEZIv/sTKcOY0xxnxa2G5zFZFo4EHgYqAC2CAiK1V1Z8Bum4FiVW0Wka8BPweu87/XoqrnhiufMcaYnoVzHMRcoFRVywFE5EngauBUgVDVtwP2XwfcFMY8xgw6j1fZdaSR9fvryEiKZcHkDMakxjsdy5iQhLNA5ACHAl5XAPN62P824K8Br+NFpARwAz9V1ee6HiAitwO3A+Tl5fU7sDEDadeRRr7++CbKq09+Yvt1xbn889UziI+JdiiZMaEJZ4GQINuCrk4kIjcBxcDSgM15qlopIgXAWyLyoaqWfeLDVB8CHgIoLi62lY9MxFi5tZLvPbOV1PgY/vXaczivMIPaE+08t/kwv3t/Px8ebuDhW4oZNyrB6ajGdCucBaICyA14PQGo7LqTiFwE3A0sVdW2zu2qWun/WS4i7wAzgbKuxxsTaUo+quM7T21hVl4aD35hFlkpcQCMG5XAmTmjOK8wk288sZk7/rCRp/52gV1JmIgVzruYNgBTRCRfRGKB64FP3I0kIjOB3wIrVPVYwPY0EYnzP88EziOg78KYSFXV2MrXHttEbnoiv7ul+FRxCLRsWjYPfP4ctlY0cM/zO7Blf02kCluBUFU3cCfwKrALeFpVd4jIvSKywr/b/UAy8Kcut7NOB0pEZCvwNr4+CCsQJqKpKv/wp62caHXzm5tmkxof0+2+l8wYy53LCnmq5BAvfXhkEFMaE7qwzuaqqi8DL3fZ9qOA5xd1c9wa4KxwZjNmoL2x6xjv7avhR1cWccbYlF73//bFU3lz9zHue3k3F00fY01NJuLYSGpjBkCb28P/e2knhdnJfHHBxN4PAKKjhHuuKuJwfQsPrSoPc0Jj+s4KhDED4NE1B/iotpkfXllETHTo/1nNL8jg8rPG8p/vlFLV2BrGhMb0nRUIY/qpze3hoffKWVSYydKpQRfm6tFdy6fT7vbyyPv7w5DOmNNnBcKYfnpu82Gqm9r42vmTT+v4vIxErjh7PI+tP0hja8cApzPm9FmBMKYfvF7lt6vKmTE+lYWTM077c/52SQEn2tw8vv7gAKYzpn+sQBjTD2/sqqK8+iR/u3QyIsEmDwiNbwBdBo+8v582t2cAExpz+qxAGNMPf1x/kPGj4rn8zLH9/qyvLC7gWFMbb+w81vvOxgwCKxDGnKaK4828t6+aa4tzcfXhzqXuLJmSxfhR8TxVcqj3nY0ZBFYgjDlNz2ysAODa4gkD8nnRUcLninN5b181FcebB+QzjekPKxDGnAaPV/lTSQWLCjOZkJY4YJ977WxfseksPsY4yQqEMadhdWkNh+tbuG5Obu8790FueiKLCjP5U0kFXq9N4mecZQXCmNPw3JbDpMa7uLhozIB/9udmT+BwfQslB44P+Gcb0xdWIIzpo9YOD6/tqGL5mWOJcw38BHu+ifuieGHrp5ZPMWZQWYEwpo/e3VvNiTY3V549PiyfnxTn4sJpY3j5wyO4Pd6wfIcxobACYUwfvbC1kvSk2H6NnO7NVeeMo/ZkO+vK68L2Hcb0xgqEMX3Q3O7mzV3HuOzMsQMy9qE755+RTVJstDUzGUdZgTCmD97afYyWDk/Ympc6xcdEc8mMsfx1+xE6rJnJOMQKhDF98NqOKjKSYpmbnx7277p0xlgaW918sN+amYwzrEAYE6IOj5e39xzjgmnZREed/sR8oVoyNZM4VxSv76wK+3cZE4wVCGNCtL68jqZWN5fM6P/EfKFIjHWxeEomr++sQtUGzZnBZwXCmBC9tvMo8TFRLCrMHLTvvLhoDIfrW9h5pHHQvtOYTlYgjAmBqvLGziqWTMkiIXbgB8d158LpYxDBmpmMI6xAGBOCHZWNVDa0hmVqjZ5kJscxOy/NCoRxhBUIY0Lwzh7fIj7LpmUP+ndfMD2bHZWNHGtqHfTvNiObFQhjQrBqbw1n5qSSmRw36N+9ZEoWAO/trRn07zYjmxUIY3rR2NrBxoPHWTo1y5HvLxrnK0zv7q125PvNyGUFwpherCmtxePVU2fygy0qSlgyNZP39lXjsTUizCCyAmFML97dW01ynItZE9Mcy7B0ahbHmzvYfrjBsQxm5LECYUwPVJVVe6tZODmDmDBOztebRYWZiMAqa2Yyg8gKhDE9KKs+yeH6Fpae4UzzUqeM5DjOyhll/RBmUFmBMKYHnWfsTvU/BFo6NYvNh+ppaOlwOooZIcJaIERkuYjsEZFSEbkryPvfEZGdIrJNRN4UkYkB790sIvv8j5vDmdOY7ry7t5qCrCRy0xOdjsLSqVl4vMqaUrvd1QyOsBUIEYkGHgQuA4qAG0SkqMtum4FiVT0beAb4uf/YdOAeYB4wF7hHRJzrITQjUmuHh/X7ayPi6gHg3NzRpMS7WLXPmpnM4AjnFcRcoFRVy1W1HXgSuDpwB1V9W1Wb/S/XARP8zy8FXlfVOlU9DrwOLA9jVmM+5YP9dbR2eB3vf+jkivZNFPjunmqb3dUMinAWiBzgUMDrCv+27twG/LUvx4rI7SJSIiIl1dV2VmUG1rt7q4l1RTE/P3xrT/fVkqlZVDa0UnrshNNRzAgQzgIRbEWVoKc9InITUAzc35djVfUhVS1W1eKsrMg4yzPDx6q91czLTx/U2Vt7s8Q/mtvuZjKDIZwFogLIDXg9AfjUCuwichFwN7BCVdv6cqwx4XKkoYV9x05ETP9Dp5zRCRRmJ7Nqn3VUm/ALZ4HYAEwRkXwRiQWuB1YG7iAiM4Hf4isOxwLeehW4RETS/J3Tl/i3GTMo1pbVArCwMHKalzotKsxkw/462t1ep6OYYS5sBUJV3cCd+P6w7wKeVtUdInKviKzw73Y/kAz8SUS2iMhK/7F1wI/xFZkNwL3+bcYMirVltYxOjGH62FSno3zKgskZtHR42FpR73QUM8y5wvnhqvoy8HKXbT8KeH5RD8c+AjwSvnTGdG9teS3z8tOJigrWHeas+fkZiPgmEZwzKd3pOGYYs5HUxnRxqK6ZiuMtLCiIvOYlgFGJMZw5fhRryqwfwoSXFQhjuvi4/yHT4STdWzg5g80H62lp9zgdxQxjViCM6WJteS2ZybFMyU52Okq3FkzOoN3jZeOB405HMcOYFQhjAqgqa8tqmVeQgUjk9T90mjMpHVeUsNqamUwYWYEwJsBHtc0cbWyN2P6HTklxLmbmjWaNvznMmHCwAmFMgM6O3wWTI7tAACyYnMmHFfU0ttr03yY8rEAYE2BtWS1jUuMoyExyOkqvFk7OwKvwQbkNETLhYQXCGD9VZV15HQsivP+h08y80cS5oqyZyYSNFQhj/EqPnaDmRNuQaF4CiHNFM2dSuo2HMGETcoEQkST/IkDGDEtry31n4gsKInf8Q1cLJmew+2gTNSfaet/ZmD7qtkCISJSI3CgiL4nIMWA3cEREdojI/SIyZfBiGhN+a0pryRmdQG56gtNRQrbQf7WzrtyamczA6+kK4m1gMvB9YKyq5qpqNrAY3+pvP/Wv42DMkOf1Kuv217Jg8tDof+h0Vs4okuNc1g9hwqKnyfouUtVP3T/nn1X1WeBZEYkJWzJjBtHuo03UN3dE/PiHrlzRUczNT2edFQgTBt1eQXQWB/+CPp8gIjcH7mPMUHeq/2GIdFAHWlCQQXnNSaoaW52OYoaZUDqpfyQi/+XvpB4jIi8AV4U7mDGDaW1ZLRMzEhk/euj0P3TqLGpr7SrCDLBQCsRSoAzYArwPPK6qnwtrKmMGkcerrN9fO+SalzpNH5dKarzLCoQZcKEUiDRgHr4i0QZMlKHUi2dML3ZUNtDU6h6SzUsA0VHCvIIM1u23AmEGVigFYh3wV1VdDswBxgOrw5rKmEHUeeY9VK8gAOYXZHCgtpnK+hano5hhJJQCcZF/+U9UtUVVvwncFd5YxgyeteW1TM5KIjs13ukop62zuFkzkxlIPQ2UmwSgqge7vqeqq8RnQviiGRN+HR4vG/bXDdnmpU7TxqaQlhhz6m4sYwZCT+Mg7heRKOB5YCNQDcQDhcAy4ELgHqAi3CGNCZdtFQ2cbPewcPLQmV4jmKgoYV5+hl1BmAHVbYFQ1WtFpAj4AvBlYBzQAuwCXgL+n6rajddmSOucomL+EO5/6LRgcgav7DjKobpmctMTnY5jhoGeriBQ1Z3A3YOUxZhBt7aslmljU0hPinU6Sr8FjoewAmEGQrcFQkSu6elAVf3zwMcxZvC0uT2UHKjj+jl5TkcZEFOyk8lMjmVteS2fn5PrdBwzDPR0BdE5WjobWAi85X+9DHgHsAJhhrQtB+tp7fAO+Q7qTiK+8RBry2pR1SE16aCJTD3NxXSrqt4KKFCkqn+jqn8DzBi0dMaE0dryWkRgfv7wKBDgu931aGMrH9U2Ox3FDAOhjIOYpKpHAl5XAVPDlMeYQbO2rJYZ41MZlTh8JiW2eZnMQAqlQLwjIq+KyC3+WVxfwrdWhDFDVmuHh80H64f06OlgCjKTyE6Js/EQZkD0eBcTgKre6e+wXuzf9JCq/iW8sYwJr00HjtPuGT79D51EhAWTM1hdav0Qpv96LRBw6o4l65Q2w8aaslqio4Q5k9KdjjLgFhRk8PyWSsqqT1KYnex0HDOE9TTVxvv+n00i0hjwaBKRxlA+XESWi8geESkVkU/N3yQiS0Rkk4i4ReRzXd7ziMgW/2NlX/+HGdOTteW1nJUzipT44dP/0OlUP4Q1M5l+6ukupkX+nymqmhrwSFHV1N4+WESigQeBy4Ai4Ab/yOxAB4FbgMeDfESLqp7rf6wI8X+PMb062eZm66H6Yde81CkvPZHxo+JtGVLTbyE1MYnIOXzcB7FKVbeFcNhcoFRVy/2f8SRwNbCzcwdV/cj/nrcPmY3pl5IDx3F7ddh1UHcSEeZPzuDdPdXWD2H6pde7mETkW8Bj+AbMZQOPicg3QvjsHOBQwOsK/7ZQxYtIiYisE5HPdJPtdv8+JdXV1X34aDOSrS2rJSZaKJ6U5nSUsFlQkEHtyXb2Vp1wOooZwkK5grgNmKeqJwFE5GfAWuA/ejku2GmL9iFbnqpWikgB8JaIfKiqZZ/4MNWHgIcAiouL+/LZZgRbU1bDubmjSYwN6QJ6SPp4PEQNZ4xNcTiNGapCGQchgCfgtYfgf/y7qgACJ4SZAFSGGkxVK/0/y/FN7TEz1GON6U5DcwfbDzcM+em9ezMhLZHc9ATrqDb9Esop1P8A60Wkc+zDZ4CHQzhuAzBFRPKBw8D1wI2hhBKRNKBZVdtEJBM4D/h5KMca05N1+2vxKiwcph3UgRYUZPDaziq8XiUqyvohTN/1egWhqg8AtwJ1wHHgVlX9ZQjHuYE7gVfxrSHxtKruEJF7RWQFgIjMEZEK4FrgtyKyw3/4dKBERLbiG7X9U//U48b0y5rSGhJiopmZN3z7HzotmJxBfXMHu46GdFe6MZ8SaiPsfsDt319EZJaqburtIFV9GXi5y7YfBTzfgK/pqetxa4CzQsxmTMhWl9UyJz+dWFcoratD2/yAdapnjB/lcBozFPVaIETkx/jGKpTxcSezAheEL5YxA6+qsZXSYye4dvbIWEp93KgEJmUksq68lq8sLnA6jhmCQrmC+DwwWVXbwx3GmHDqnOF0uHdQB1owOYMXtx3B41WirR/C9FEo19nbgdHhDmJMuK0urWFUQgxF43udCGDYWDA5k6ZWN9sPNzgdxQxBoVxB3AdsFpHtQFvnRpv+wgwlqsqasloWFGSMqDPpzru13i+t4ZxcO88zfRNKgfg98DPgQ8CmxDBD0oHaZg7Xt3DH0pHVFp+ZHMf0camsLq3h68sKnY5jhphQCkSNqv572JMYE0ary2oAWFg4cvofOi0qzOD3aw7Q0u4hITba6ThmCAmlD2KjiNwnIgtEZFbnI+zJjBlAa8pqGZMaR0FmktNRBt2iKVm0e7x88FGd01HMEBPKFUTnFBfzA7bZba5myPB6lbVltZw/NWtEzmw6d1I6sdFRrC6tYenULKfjmCEklCVHlw1GEGPCZffRJupOto/I5iWAhNhoZk9M4719NU5HMUNMKAPlvhNkcwOwUVW3DHwkYwbWms7+hxEw/1J3Fk3J5P5X91Bzoo3M5Din45ghIpQ+iGLgDnxrOeQAtwPnA/8tIt8LXzRjBsaqfTVMzkpi/OgEp6M4ZpH/6ml1qV1FmNCFUiAygFmq+l1V/S6+gpEFLME3BYcxEau1w8P68lqWjPC29zNzRjEqIYb3rZnJ9EEoBSIPCJxmowOYqKotBAycMyYSfbC/jja3d8QXiOgoYeHkDFaX1qBqa2uZ0IRSIB4H1onIPSJyD7AaeEJEkghYX9qYSPTu3mpiXVHMzx+5/Q+dFk3JpLKhlfKak05HMUNEKHcx/VhEXgYW4VtJ7g5VLfG//YVwhjOmv1btrWbupHQbIAYsLvRdRb2/r4bJWckOpzFDQbdXECKS6v+Zjm89iD8AjwLl/m3GRLTK+hb2HTvBkqkj8/bWrvIyfMuQvm8d1SZEPV1BPA5cCWzk43UgwHcVocDImtTGDDnv7asGGPH9D4EWFWbx4tZKOjxeYqKH/6JJpn+6/Q1R1Sv9P/NVtSDgka+qVhxMxFu1t4YxqXGcMSbF6SgRY+nUTJra3Gw6cNzpKGYI6PUUQkTO83dIIyI3icgDIpIX/mjGnD6PV3m/tIbFU0bm9BrdOa8wE1eU8PaeaqejmCEglGvM/wKaReQc4HvAAXz9EcZErK0V9TS0dFjzUhcp8THMmZTOO3uOOR3FDAGhFAi3+m6cvhr4lar+CrBrdhPRVu2tRgQWj9D5l3qybFoWu482UVnf4nQUE+FCKRBNIvJ94CbgJRGJBmLCG8uY/lm1t5qzc0aRlhTrdJSIs+yMbADesWYm04tQCsR1+EZM36aqR/HNx3R/WFMZ0w8NzR1sOVRvzUvdKMxOJmd0Am9bM5PpRSgD5Y4CDwS8PohvPIQxEen90hq8are3dkdEWDYtiz9vOkyb20OcywYRmuDsRmgz7Ly5q4rRiTHMzB3tdJSIteyMbJrbPWzYb7e7mu5ZgTDDitvj5e09x1h2RjYuGwjWrQWTM4h1RVkzk+lRT1NtvOn/+bPBi2NM/2w6WM/x5g4umj7G6SgRLTHWxfyCDCsQpkc9nWKNE5GlwAoRmSkiswIfgxXQmL54c1cVMdFi8y+FYNkZWZRXn+RArc3uaoLrqUD8CLgLmICvk/oXAY9/DX80Y/rujV1VzMvPICXe7sTujd3uanrT01xMz6jqZcDPVXVZl8cFg5jRmJDsrzlJWfVJLpqe7XSUIWFSZhIFmUm8tduamUxwoa4HsQLfEqMA76jqi+GNZUzfvbmrCoALrf8hZMumZfOHtQc40eYmOa7XPwdmhAllsr77gG/hWz1uJ/At/7ZeichyEdkjIqUicleQ95eIyCYRcYvI57q8d7OI7PM/bg7tf44ZyV7fWcW0sSnkpic6HWXIWH7mWNo9XpubyQQVyn2AVwAXq+ojqvoIsNy/rUf+KTkeBC4DioAbRKSoy24HgVvwrT0ReGw6cA8wD5gL3CMiaSFkNSNUQ3MHJQeOc6E1L/XJrLw0MpNjeWX7UaejmAgU6o3igSOORoV4zFygVFXLVbUdeBLfhH+nqOpHqroN8HY59lLgdVWtU9XjwOv4CpMxQb2z9xger1rzUh9FRwkXF43h7d3HaO3wOB3HRJhQCsR9wGYR+V8R+T2+FeZ+EsJxOcChgNcV/m2hCOlYEbldREpEpKS62u7EGMle31lFZnIs506w0dN9demMsZxs97CmzJYiNZ/Ua4FQ1SeA+cCf/Y8FqvpkCJ8dbJUWDbLttI9V1YdUtVhVi7OybN6dkarD4+XdvdVcMC2bqChbHKivFk7OJCXOZc1M5lNCum1BVY8AK/v42RVAbsDrCUBlH449v8ux7/Tx+80IsaaslqZWt42ePk2xrigumJ7NG7uO4fZ4bYoSc0o4fxM2AFNEJF9EYoHrCb3IvApcIiJp/s7pS/zbjPmUl7ZVkhznstlb++HSGWOpO9nOho9s8j7zsbAVCFV1A3fi+8O+C3haVXeIyL3+cRWIyBwRqQCuBX4rIjv8x9YBP8ZXZDYA9/q3GfMJ7W4vr+6o4uKiMcTH2LTVp2vp1CziXFG8usOamczHum1i8t9q2q1Q/mCr6svAy122/Sjg+QZ8zUfBjn0EeKS37zAj2+qyGhpaOrjy7HFORxnSkuJcLJ6SxWs7jnLPVUWIWF+O6fkKYiNQ4v9ZDewF9vmfbwx/NGN699K2I6TEu1g0xSbn66/lZ46lsqGVDw83OB3FRIie5mLKV9UCfE1EV6lqpqpmAFfiu5vJGEe1uT28uuMolxSNtVXRBsBF07OJjhK7m8mcEkofxBx/UxEAqvpXYGn4IhkTmvf31dDU6rbmpQEyOjGWhZMzeHHbEVRDvSPdDGehFIgaEfmBiEwSkYkicjdQG+5gxvTmpW1HSI13cV6hNS8NlKvPzeFgXTObDtY7HcVEgFAKxA1AFvAX4Dkg27/NGMe0dnh4fWcVl84YS6zL7tsfKJfOGEOcK4rntxx2OoqJAKGMpK5T1W+p6kz/41t2y6lx2nv7amhqc3OFNS8NqJT4GC4uGsOL247Q4ek6RZoZaUKZ7jtLRO4XkZdF5K3Ox2CEM6Y7L26rZFRCjDUvhcFnzs2h7mQ77+2z+c1GulCuzR8DdgP5wD8DH+EbvGaMI1raPbyxs4rlM8YSY9NCDLglU7MYnRjDc5tDnRnHDFeh/NeVoaoPAx2q+q6qfhnf5H3GOOK1nUc52e7hMzNDnRzY9EWsK4orzx7HazuPcqLN7XQc46BQCkSH/+cREblCRGbSzehnYwbDs5sOkzM6gXn5PQ72N/3wmXNzaO3w8ppNvTGihVIg/kVERgHfBf4B+B3w7bCmMqYbVY2tvL+vms/OzLGpvcNo9sQ0JqQl8NwWa2YayXosEP5lQ6eoaoOqblfVZao6W1X7OvW3MQPiuc2H8SpcM8ual8JJRPjMuTm8v6+aY02tTscxDumxQKiqB1gxSFmM6ZGq8uymCmbmjaYgK9npOMPeZ2aOx6vwwtYjTkcxDgmliWmNiPxaRBaLyKzOR9iTGdPFlkP17K06wbWzc3vf2fRbYXYK50wYxdMbDtnUGyNUKCvKLfT/vDdgmwIXDHwcY7r35AeHSIyNZsW5452OMmJcPzeP7//5QzYfqmdWXprTccwg67VAqOqywQhiTE9OtLl5YVslV549juS4kFbKNQPgqnPG8+MXd/LkBwetQIxAvf6XJiLfCbK5AdioqlsGPpIxn/bC1kqa2z1cPzfP6SgjSnKci6vOHs/KrZX88MoiUuJjnI5kBlEofRDFwB1Ajv9xO3A+8N8i8r3wRTPmY098cJAzxqQwM3e001FGnOvn5tLS4eF5u+V1xAlpJDUwS1W/q6rfxVcwsoAlwC1hzGYM4Ouc3lbRwBfm59lSmA44N3c0ReNS+cPaA9ZZPcKEUiDygPaA1x3ARFVtAdrCksqYAI+u+YjkOBfXzLIB/E4QEW5eOJE9VU18sN8mch5JQikQjwPrROQeEbkHWA08ISJJwM6wpjMjXs2JNl7cdoS/mZVjndMOWnFODqMSYnh07QGno5hBFMp6ED8GvgrU4+ucvkNV71XVk6r6hXAHNCPbUxsO0e7x8sUFk5yOMqIlxEZz3ZxcXtlxlKMNNrJ6pAhprmRV3aiqv1LVX6pqSbhDGQPQ7vby6NqPWFSYSWG2jZx22k3zJuJV5dG1HzkdxQwSm0zfRKwXtlZS1djGVxbnOx3FAHkZiSyfMZY/rjvASZsGfESwAmEikqry3++VM3VMMkunZjkdx/h9ZXEBja1u/lRyyOkoZhBYgTARaXVpLbuPNvGVxQV2a2sEmT0xjdkT03h49X7ctmb1sGcFwkSk37xbRlZKHFfbvEsR56uLCzhU18JLH9osr8OdFQgTcTYdPM77pTXcvriAOFe003FMF5cUjaEwO5n/fLsMr9cGzg1nViBMxPmPN/eRlhjDjfNs3qVIFBUl3LmskD1VTby2s8rpOCaMrECYiLL9cANv76nmK4sLSLKBcRHryrPHMSkjkV+/vc+m3xjGwlogRGS5iOwRkVIRuSvI+3Ei8pT//fUiMsm/fZKItIjIFv/jN+HMaSLHL9/YS0q8iy8umOh0FNMDV3QUf7eskO2HG3ndriKGrbAVCP961g8ClwFFwA0iUtRlt9uA46paCPwb8LOA98pU9Vz/445w5TSRY+OB47yx6xh3LJ1Mqk0rHfGumZlDfmYSv3htr/VFDFPhvIKYC5SqarmqtgNPAld32edq4Pf+588AF4rd0zgiqSr3v7qbzORYbj1vktNxTAhc0VF85+Kp7KlqYuVWmwp8OApngcgBAkfTVPi3Bd1HVd345nrK8L+XLyKbReRdEVkcxpwmAry3r4Z15XXcuayQxFjrexgqrjhrHEXjUnng9b20u21cxHATzgIR7Eqg63Vod/scAfJUdSbwHeBxEUn91BeI3C4iJSJSUl1d3e/Axhker/KTl3cxIS2BG+zOpSElKkr4x+VncLCu2eZoGobCWSAqgNyA1xOArtehp/YRERcwCsiY8QIAABDYSURBVKhT1TZVrQXfRIFAGTC16xeo6kOqWqyqxVlZNh3DUPXUhkPsPtrEP10+3cY9DEHnT81iydQs/v3NfRw/2d77AWbICGeB2ABMEZF8EYkFrgdWdtlnJXCz//nngLdUVUUky9/JjYgUAFOA8jBmNQ5pbO3gF6/tYe6kdC47c6zTccxpEBF+cMV0TrZ7+OUbe52OYwZQ2AqEv0/hTuBVYBfwtKruEJF7RWSFf7eHgQwRKcXXlNR5K+wSYJuIbMXXeX2HqtpSVsPQL1/fR11zOz+4crrNuTSETR2Two1z8/jj+oPsrGx0Oo4ZIDJcBrkUFxdrSYktVTGUbD/cwIpfv8/1c/P4yWfPcjqO6af65nYu/MW75GUk8uwdC4mKsoI/FIjIRlUtDvaejaQ2jvB6lR88t520xFj+z6XTnI5jBsDoxFj+6fLpbD5Yz5MbbDrw4cAKhHHEo2s/Ysuheu6+YjqjEm1Q3HBxzawc5uWnc99fd9nSpMOAFQgz6A7UnuRnr+zh/DOy+OzMrkNjzFAmIvz0b86mw+Pl+3/eZvM0DXFWIMyg8nqVf3xmG65o4b5rzrKO6WEoPzOJ7106jbf3VPPMxgqn45h+sAJhBtVD75Xzwf46fnhlEeNGJTgdx4TJLQsnMXdSOve+sJODtc1OxzGnyQqEGTRbDtXzr6/u4fKzxnLt7AlOxzFhFBUlPHDdOSDwzSc302HLkw5JViDMoGho6eCbT2xmTGo89332bGtaGgEmpCXy02vOZsuhen7xmg2gG4qsQJiw83qV7zy1hcr6Fv79hnPtrqUR5Iqzx3HjvDx+824Zr2w/6nQc00dWIEzY/cdbpby5+xg/vLKI2RPTnY5jBtk9VxVxTu5ovvv0FkqPNTkdx/SBFQgTVi9tO8K/vbGXa2bl8CVbJW5EinNF85ubZpEQG81Xfl9CnU3oN2RYgTBhs/HAcb799BaKJ6bxk8/aLa0j2bhRCTz0pWKONLTy1UdLaO3wOB3JhMAKhAmLfVVNfPXREsaPiuehLxUTH2PTeI90s/LS+LfrzmXjgeN868nNuO3OpohnBcIMuIO1zdz08Hqio4T/vXUu6UmxTkcyEeLys8Zxz1VFvLqjiu89u83Wso5wtrajGVCH6pq58XfraHN7eer2BUzKTHI6kokwt56XT1Ormwde30tsdBQ/+exZNvNrhLICYQbM/pqT3Pjf62hu9/CH2+ZyxtgUpyOZCPWNCwppc3t48O0y2j1e7v/cOURbkYg4ViDMgNhWUc+X/3cDXoUnvjqfovGfWkLcmFNEhH+8dBpxrmgeeH0vDc0d/OqGmSTH2Z+kSGJ9EKbf3t59jOsfWkecK5qn/3aBFQcTsm9eOIUfXz2Dd/ZWc+1v1lJZ3+J0JBPACoQ5barKf71Txpd/v4H8zCT+8vWFFGYnOx3LDDFfXDCJR26ZQ0VdM1c/uJqth+qdjmT8rECY03L8ZDt3/HEjP3tlN1ecNY5n7lhIdkq807HMELV0ahbP/t1C4lxRfP63a3ls/QFbSyICWIEwfbZqbzWX/nIVb+0+xt2XT+c/bphJQqyNczD9M3VMCs99/Tzm5qdz91+289VHS6g50eZ0rBHNCoQJWWuHh39+YQdfeuQDUhNieO7r5/HVJQU2QtoMmMzkOH5/61x+eGURq/bVsPyXq3hrd5XTsUYsu2XA9EpVeWX7Uf7lpV0crm/hloWTuOuyaTY62oRFVJRw26J8zivM4O+f3MKX/7eES2eM4QdXFJGbnuh0vBFFhks7X3FxsZaUlDgdY9jZW9XE/125gzVltZwxJoV7VhSxcHKm07HMCNHa4eHh9/fz67dK8ahyx5ICvnZ+oTVpDiAR2aiqxUHfswJhgik9doL/fKeU57dUkhzn4ruXTOXGuXm4oq1V0gy+Iw0t3PfyblZurSQ7JY7blxRw47w8EmOtEaS/rECYkO2sbOTBd0p5+cMjxLuiuXFeHl9fVmjzKZmIsOGjOh54bS9ry2tJT4rltkX5fHHBRFLjbRGq02UFwvSotcPDS9uO8MQHByk5cJzkOBc3L5zIl8/LJyM5zul4xnxKyUd1/PrtUt7ZU01CTDQrzhnPDfPyOGfCKLtpoo+sQJhP8XiVko/qeOnDIzy/pZKGlg7yM5O4YW4u1xXn2bKgZkjYfriBP6w9wMqtlbR0eCgal8o1s3K47Kxx5IxOcDrekGAFwgC+K4UP9tfx+s4qXtlxlOqmNuJcUVxcNIYb5+WxoCDDzr7MkNTU2sHzWyp5csNBth9uBOCc3NFcfuZYLpiWTWF2sv1ud8MKxAjV7vay60gj6/fX8t6+Gtbvr6Pd7SU+JooLpmVz2ZnjWDYt2yZIM8PKRzUneXn7Ef764VE+PNwAwJjUOBZPyWLxlExmT0wjZ3SCFQw/KxAjQIfHS3n1SXYfbWRnZSObDh5nW0UDbW7fql1TxySzqDCLxVMzmZefbnd/mBHhcH0L7+2t5r19Nawuq6G+uQOA7JQ4ZuWlcW7eaKaNTWHa2FTGpMaNyKJhBWKYUFWON3dwqK6Zg/7H3qom9hxtoqz6BB0e379lTLQwY/woZk9MY1ZeGsWT0hiTavMkmZHN41V2HfGdPG08cJxNB49zqO7j2WNHJcRwxtgUpo1NoTA7mdy0RHLTE5iQljisB4U6ViBEZDnwKyAa+J2q/rTL+3HAo8BsoBa4TlU/8r/3feA2wAN8U1Vf7em7hnKBUFVOtnuobmqj5kQb1U1tn3p+uL6FQ3XNnGz/5GLvOaMTOGNsiu8xxvezICuJONfw/YU2ZqAcP9nOHv9J1u6jTew52sjeqhOcaHN/Yr/slDhy0xMZmxpPVkocWSlxZCbH+p4n+7ZlJMcSMwTHCfVUIMLWziAi0cCDwMVABbBBRFaq6s6A3W4DjqtqoYhcD/wMuE5EioDrgRnAeOANEZmqqp/86zhIVJUOj+L2eulwKx1eL26P0uHx0uHx0ub20tLhoaXd/+jwfPw64GdTaweNLW4aWztobOmgsdXt/9lx6uw/UJRARnIcmclxTEhLYH5BBrnpieSl+85sctMSSbL+A2NOW1pSLPMLMphfkHFqm6pSfaKNQ3XNHKprOXXFfuh4M7uONrJqXxtNre6gn5cYG01qfAypCS7/zxhS412kJsSQFOciMSaahNho4mOiSYyNJiEmmnj/z0T/9tjoKGJcUcRECTH+564oITY6atCXZg3nX5e5QKmqlgOIyJPA1UBggbga+L/+588AvxZfI+DVwJOq2gbsF5FS/+etHeiQx0+28/nfrsXtVdrdXl8RCPjj7/Yo7gFYWD0+JuoTvzBpSbHkZSSd+uUZnRDjPyuJO3WGkpYYa8swGjPIRITslHiyU+KZPTH4Pq0dXa74T7RRe6L91Alf54lgdVMbZdUnaGzp4ESbO+iJYF9ER8mpYtFZOGKiozgrZxS/+eLsfn12MOEsEDnAoYDXFcC87vZRVbeINAAZ/u3ruhyb0/ULROR24HaAvLy80woZ44qiMDuZmOgoXNG+/+Nd0f7KHR1FTLTgiooiNuAfI8b/vsv/PM4VRUKsi4QY35lAQuwnX8e5Br/yG2PCJz4mmtz0xD5PHtjh8dLqb2FobffS3OE+1cLQ2uGhud3jPzn1n6S6/c87Wy/8J66BJ7EdHmVCWnjGfISzQAT7i9i1fHa3TyjHoqoPAQ+Brw+irwEBkuNc/NdNA195jTGmq84Tz5QhMjVIOHtUKoDcgNcTgMru9hERFzAKqAvxWGOMMWEUzgKxAZgiIvkiEouv03lll31WAjf7n38OeEt9t1WtBK4XkTgRyQemAB+EMasxxpguwtbE5O9TuBN4Fd9tro+o6g4RuRcoUdWVwMPAH/yd0HX4igj+/Z7G16HtBr7u1B1MxhgzUtlAOWOMGcF6Ggcx9EZ1GGOMGRRWIIwxxgRlBcIYY0xQViCMMcYENWw6qUWkGjjQj4/IBGoGKM5Aslx9F6nZLFffRGouiNxsp5NroqpmBXtj2BSI/hKRku568p1kufouUrNZrr6J1FwQudkGOpc1MRljjAnKCoQxxpigrEB87CGnA3TDcvVdpGazXH0TqbkgcrMNaC7rgzDGGBOUXUEYY4wJygqEMcaYoKxA+InIuSKyTkS2iEiJiMx1OlMgEfmGiOwRkR0i8nOn8wQSkX8QERWRTKezAIjI/SKyW0S2ichfRGS0w3mW+//tSkXkLiezBBKRXBF5W0R2+X+vvuV0pkAiEi0im0XkRaezdBKR0SLyjP/3a5eILHA6E4CIfNv/b7hdRJ4QkfiB+FwrEB/7OfDPqnou8CP/64ggIsvwrdN9tqrOAP7V4UiniEgucDFw0OksAV4HzlTVs4G9wPedCiIi0cCDwGVAEXCDiBQ5lacLN/BdVZ0OzAe+HkHZAL4F7HI6RBe/Al5R1WnAOURAPhHJAb4JFKvqmfiWV7h+ID7bCsTHFEj1Px9FZK1g9zXgp6raBqCqxxzOE+jfgO8RZElYp6jqa6rq9r9ch29FQqfMBUpVtVxV24En8RV7x6nqEVXd5H/ehO+P3afWfneCiEwArgB+53SWTiKSCizBt44NqtquqvXOpjrFBST4V+ZMZID+flmB+NjfA/eLyCF8Z+iOnXUGMRVYLCLrReRdEZnjdCAAEVkBHFbVrU5n6cGXgb86+P05wKGA1xVEyB/hQCIyCZgJrHc2ySm/xHfi4XU6SIACoBr4H3/T1+9EJMnpUKp6GN/frIPAEaBBVV8biM8O24pykUhE3gDGBnnrbuBC4Nuq+qyIfB7fWcJFEZLNBaThawaYAzwtIgU6CPco95Lrn4BLwp0hmJ5yqerz/n3uxteM8thgZutCgmyLmKstABFJBp4F/l5VGyMgz5XAMVXdKCLnO50ngAuYBXxDVdeLyK+Au4AfOhlKRNLwXZXmA/XAn0TkJlX9Y38/e0QVCFXt9g++iDyKr80T4E8M8qVtL9m+BvzZXxA+EBEvvkm5qp3KJSJn4fuF3Coi4GvG2SQic1X1qFO5AvLdDFwJXDgYhbQHFUBuwOsJRFDzpYjE4CsOj6nqn53O43cesEJELgfigVQR+aOq3uRwrgqgQlU7r7KewVcgnHYRsF9VqwFE5M/AQqDfBcKamD5WCSz1P78A2Odglq6ew5cJEZkKxOLwTJKq+qGqZqvqJFWdhO8/nlmDURx6IyLLgf8DrFDVZofjbACmiEi+iMTi6zxc6XAmAMRX2R8GdqnqA07n6aSq31fVCf7fq+uBtyKgOOD/3T4kImf4N10I7HQwUqeDwHwRSfT/m17IAHWej6griF58FfiVv5OnFbjd4TyBHgEeEZHtQDtws8NnxZHu10Ac8Lr/6madqt7hRBBVdYvIncCr+O4ueURVdziRJYjzgC8CH4rIFv+2f1LVlx3MFOm+ATzmL/blwK0O58Hf3PUMsAlfk+pmBmjKDZtqwxhjTFDWxGSMMSYoKxDGGGOCsgJhjDEmKCsQxhhjgrICYYwxJigrEMYYY4KyAmGMMSYoKxDGhImIzPGvSREvIkn++frPdDqXMaGygXLGhJGI/Au++YQS8M3jc5/DkYwJmRUIY8LIPyXDBnzTtyxUVY/DkYwJmTUxGRNe6UAykILvSsKYIcOuIIwJIxFZiW8VuXxgnKre6XAkY0Jms7kaEyYi8iXAraqP+9emXiMiF6jqW05nMyYUdgVhjDEmKOuDMMYYE5QVCGOMMUFZgTDGGBOUFQhjjDFBWYEwxhgTlBUIY4wxQVmBMMYYE9T/B872BYAzbhQWAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "x.grad.zero_()\n",
    "y.sum().backward()\n",
    "xyplot(x, x.grad, 'grad of sigmoid')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_a4n2fep",
    "id": "71DDEDEDFC494B4E912AC070F4A2E9A9",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "#### tanh函数\n",
    "tanh（双曲正切）函数可以将元素的值变换到-1和1之间：\n",
    "\n",
    "\n",
    "$$\n",
    "\\text{tanh}(x) = \\frac{1 - \\exp(-2x)}{1 + \\exp(-2x)}.\n",
    "$$\n",
    "\n",
    "\n",
    "我们接着绘制tanh函数。当输入接近0时，tanh函数接近线性变换。虽然该函数的形状和sigmoid函数的形状很像，但tanh函数在坐标系的原点上对称。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "graffitiCellId": "id_m4ur5k0",
    "id": "92D16076309F42169482834C0B6ABB24",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZAAAAEGCAYAAABLgMOSAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXxc9X3u8c8jyZb33djGu4nZjMEG4YBpFhYnhvbi7CFpqLPVaW7I1qYNJLdJLmka0tyWpm1uE5fQkoRACkmKb4AQ1qQJSy0bY8s2jheMJctgeZG8yJYs6Xv/mGM6CMmWJY3OjOZ5v17zmnN+Z5nHMNJX5/zOOT9FBGZmZqeqJO0AZmZWmFxAzMysW1xAzMysW1xAzMysW1xAzMysW8rSDtCXxo0bFzNmzEg7hplZQVm1atWeiBjfvr2oCsiMGTOorKxMO4aZWUGR9GJH7T6FZWZm3eICYmZm3eICYmZm3eICYmZm3eICYmZm3ZJqAZF0u6Tdkqo6WS5J/yBpi6S1ki7MWrZU0ubktbTvUpuZGaR/BPJvwOITLL8amJ28lgH/DCBpDPBl4PXAAuDLkkbnNKmZmb1KqveBRMSvJc04wSpLgO9H5pnzT0saJWkS8Gbg4YjYByDpYTKF6K7cJjaznogImlraaGxupbG5hSPNrTQ2t3L0WCutbcGxtqC1rY2W1qC1LWhpC1razbe2BW0RREAk+8zsO/kMspdl5l+1PGtbstbJbuuj/xh9+WksXTiDscPKe3Wf+X4j4WSgOmu+JmnrrP01JC0jc/TCtGnTcpPSrIhFBHWHmtixt5Htext5+cBR9h5qZu/hJvYeambPoSb2HW7mcFMLR4610uYhiF4h9d1nXTtvctEVkI7+88YJ2l/bGLEcWA5QUVHhr65ZD7S1BVvqDvFcdT3rdjbwXE0Dm18+SGNz66vWGzqwlLHDyhk7bCBTRg/m/CkjGVY+gCEDSxk8sJQhyWvwwDKGDMi0lZWIslJRWlLyynRZyavnS0tEqTLvAEKgzC9iAdLx9uNtetUv6ey2Dtfvy9/o/UC+F5AaYGrW/BSgNml/c7v2J/oslVkROdLcym+37OHR51/mkY27qTvYBGSKxHmTR/KeiqnMGDuE6eOGMn3MEE4fNZhBA0pTTm19Id8LyArgBkl3k+kwb4iIXZIeAv46q+P8LcBNaYU0649W79jPnU/v4P51tRw91saw8jLedOZ43nzWeOZPG82scUMpKfFf7MUs1QIi6S4yRxLjJNWQubJqAEBEfAd4ALgG2AI0Ah9Klu2T9FVgZbKrm493qJtZz/xm8x7+4dHN/Nf2fQwdWMrb50/hmrkTef3MsQwsS/vCTcsnij6+EiBNFRUV4afxmnVs+57D3PzzDTz2/G4mjhjEx940i3dXTGVYeb6fqLBck7QqIirat/ubYVbkIoI7ntzOXz/4PANKxBeuOZulC2dQXuZ+DDsxFxCzInbg6DE+c/caHnt+N1ecfRq3vGMup40YlHYsKxAuIGZFqnpfIx+5YyXb6g5z85I5XH/JdF/GaqfEBcSsCL2w5zDv/e5THD3Wyvc/soCFZ4xLO5IVIBcQsyKzY28j7/+Xp2ltC+79+ELOnDA87UhWoFxAzIrI/sPNXH/7Mxw51spdf3yJi4f1iC/qNisSx1rb+MSPVrOr/ijfW3ox50wakXYkK3A+AjErEt948Hme3LqXb77rfC6a7tEPrOd8BGJWBH67ZQ+3/eYFrr9kOu+umHryDcy6wAXErJ9rOHKMz93zHLPGD+UL15yTdhzrR3wKy6yf+/oDG9l9sImffnwhgwf67nLrPT4CMevH1tbU8+PKaj582QwumDoq7TjWz7iAmPVTbW3BV1asZ+zQcj515ey041g/5AJi1k+teK6W1Tvq+fzisxg+aEDacawfcgEx64daWtu49ZHfMef0Ebzzwilpx7F+ygXErB+6b00tL+5t5NNXzvaogZYzqRYQSYslbZK0RdKNHSy/VdKa5PU7SfVZy1qzlq3o2+Rm+aultY1/enwL504awaJzJ6Qdx/qx1C7jlVQKfBtYBNQAKyWtiIgNx9eJiM9mrf9JYH7WLo5ExLy+ymtWKO5ft4sX9hzmu9df5MezW06leQSyANgSEdsiohm4G1hygvXfB9zVJ8nMClRE8L3fvMAZ44ey6BwffVhupVlAJgPVWfM1SdtrSJoOzAQey2oeJKlS0tOS3tbZh0halqxXWVdX1xu5zfLWs9X1rK1p4IMLZ7jvw3IuzQLS0bc7Oln3OuDeiGjNapuWDPL+fuDvJZ3R0YYRsTwiKiKiYvz48T1LbJbn7nhyO8PLy3iHr7yyPpBmAakBsp/qNgWo7WTd62h3+ioiapP3bcATvLp/xKzo7D5wlPvX7uLdFVMZWu6nFFnupVlAVgKzJc2UNJBMkXjN1VSSzgJGA09ltY2WVJ5MjwMuAza039asmNyzqoaWtuD6S6enHcWKRGp/pkREi6QbgIeAUuD2iFgv6WagMiKOF5P3AXdHRPbprXOA70pqI1MEb8m+esus2EQEP1ldw4IZY5g5bmjacaxIpHqcGxEPAA+0a/tSu/mvdLDdk8DcnIYzKyBrquvZVneYj71xVtpRrIj4TnSzfuCnq3cyaEAJ18ydlHYUKyIuIGYFrqmllRXP1fLWORP90ETrUy4gZgXuV5vqaDhyzJfuWp9zATErcA9WvcSoIQNYeMbYtKNYkXEBMStgzS1tPLLxZRadM4EBpf5xtr7lb5xZAfvt1j0cPNrC1XMnph3FipALiFkBe3DdLoaXl3HZ68alHcWKkAuIWYFqaW3j4Q0vc8U5p1FeVpp2HCtCLiBmBaryxf3sbzzGW+f49JWlwwXErEA9vmk3ZSXiDbN9+srS4QJiVqB+tamOi2eM8c2DlhoXELMCVFt/hOdfOsjlZ3uMG0uPC4hZAXpiU2Z0zTefdVrKSayYuYCYFaAnNu1m8qjBzD5tWNpRrIi5gJgVmOaWNn67ZQ9vPms8ksc9t/S4gJgVmDXV9RxubuWNZ7r/w9KVagGRtFjSJklbJN3YwfIPSqqTtCZ5fTRr2VJJm5PX0r5NbpaeJ7fuoURwySw/PNHSldqIhJJKgW8Di4AaYKWkFR0MTfvjiLih3bZjgC8DFUAAq5Jt9/dBdLNUPbl1L+dNHsnIwb5819KV5hHIAmBLRGyLiGbgbmBJF7d9K/BwROxLisbDwOIc5TTLG43NLTy7Yz+X+tHtlgfSLCCTgeqs+Zqkrb13Slor6V5JU09xWyQtk1QpqbKurq43cpulpnL7fo61BgvP8N3nlr40C0hHl49Eu/n/B8yIiPOBR4A7TmHbTGPE8oioiIiK8ePd6WiF7cmtexlQKi6eMTrtKGapFpAaYGrW/BSgNnuFiNgbEU3J7L8AF3V1W7P+6Kmte5g/dTRDBqbWfWn2ijQLyEpgtqSZkgYC1wErsleQNClr9lpgYzL9EPAWSaMljQbekrSZ9VsHjx5j3c4GLnH/h+WJ1P6MiYgWSTeQ+cVfCtweEesl3QxURsQK4FOSrgVagH3AB5Nt90n6KpkiBHBzROzr83+EWR9a9eJ+2gIWzBiTdhQzIMUCAhARDwAPtGv7Utb0TcBNnWx7O3B7TgOa5ZHK7fspLRHzp41KO4oZ4DvRzQrGyu37mHP6CIaWu//D8oMLiFkBaGppZU11PRXTffrK8ocLiFkBqNp5gKaWNhbM9OW7lj9cQMwKQOX2zDUiF/kIxPKIC4hZAVi5fR8zxw1l/PDytKOYvcIFxCzPRQSrXtxPxXSfvrL84gJilud27Gtkf+Mx5k9zAbH84gJilufWVNcDMG+q7/+w/OICYpbn1lTXM2hACWdO8Pjnll9cQMzy3HPV9cydPJKyUv+4Wn7xN9Isjx1rbaOq9gAXTPHpK8s/LiBmeez5XQdpbmljnp9/ZXnIBcQsj62pyXSg+wjE8pELiFkee666nrFDBzJl9OC0o5i9hguIWR57rrqeeVNHIXU0irNZulItIJIWS9okaYukGztY/qeSNkhaK+lRSdOzlrVKWpO8VrTf1qzQHTx6jC11h7jA939YnkptYAFJpcC3gUVkxjhfKWlFRGzIWu1ZoCIiGiV9HPgb4L3JsiMRMa9PQ5v1oXU1DUTgAmJ5K80jkAXAlojYFhHNwN3AkuwVIuLxiGhMZp8GpvRxRrPU/HcH+siUk5h1LM0CMhmozpqvSdo68xHgwaz5QZIqJT0t6W2dbSRpWbJeZV1dXc8Sm/WhNTvqmTluKKOGDEw7ilmH0hwbs6NewehwRekDQAXwpqzmaRFRK2kW8JikdRGx9TU7jFgOLAeoqKjocP9m+ei5mnounTU27RhmnUrzCKQGmJo1PwWobb+SpKuALwLXRkTT8faIqE3etwFPAPNzGdasL73UcJSXDzS5/8PyWpoFZCUwW9JMSQOB64BXXU0laT7wXTLFY3dW+2hJ5cn0OOAyILvz3ayg+Qm8VghSO4UVES2SbgAeAkqB2yNivaSbgcqIWAF8ExgG3JNcB78jIq4FzgG+K6mNTBG8pd3VW2YFbU11PQNKxTmTRqQdxaxTafaBEBEPAA+0a/tS1vRVnWz3JDA3t+nM0rO2pp6zJ45g0IDStKOYdcp3opvlmYigamcDc335ruU5FxCzPFO97wgHjrZw3ukuIJbfXEDM8kxVbQMAcye7gFh+cwExyzNVOxsoKxFnTvQQtpbfutSJLqkCeANwOnAEqAIeiYh9OcxmVpSqag9w5oThlJe5A93y2wmPQCR9UNJq4CZgMLAJ2A38HvCwpDskTct9TLPiEBGs39nAeZN9+a7lv5MdgQwFLouIIx0tlDQPmA3s6O1gZsXopQNH2Xu4mfPc/2EF4IQFJCK+3dkySQMjYk3vRzIrXlU7DwAwx1dgWQHoUie6pCckzciaX0DmUSRm1ouqdjZQIjhn0vC0o5idVFfvRP868AtJ/0DmketXAx/KWSqzIrW+toEzxg9jyMBUHxJh1iVd+pZGxEOS/gR4GNgDzI+Il3KazKwIrdvZwMIzxqUdw6xLunoK6y+BfwTeCHwFeELS7+cwl1nR2X0w8wj3Oaf7CiwrDF09Th4HLEiuxnpK0i+A24D7c5bMrMisr810oPsKLCsUXT2F9el28y8Ci3KSyKxIrd+ZeYTJuT4CsQJxshsJl0vq8LHpkoZK+rCkP8xNNLPiUrXzADPGDmHEoAFpRzHrkpMdgfxf4C+TIlIF1AGDyNw8OAK4HbgzpwnNikRVbYNHILSCcsIjkIhYExHvAS4Gvg38J5lhZz8aERdExLeyxyk/VZIWS9okaYukGztYXi7px8nyZ9rdi3JT0r5J0lu7m8EsH9Q3NlOz/4j7P6ygdLUP5BDwRG9+sKRSMkVpEVADrJS0ot3QtB8B9kfE6yRdB3wDeK+kc8mMoT6HzAMeH5F0ZkS09mZGs77ySge670C3AtLVy3gvk/SwpN9J2ibpBUnbevjZC4AtEbEtIpqBu4El7dZZAtyRTN8LXKnM4OhLgLsjoikiXgC2JPszK0hVSQe6L+G1QtLVy3i/B3wWWAX01l/5k4HqrPka4PWdrRMRLZIagLFJ+9Pttp3c0YdIWgYsA5g2zQ8OtvxUVXuAyaMGM3rowLSjmHVZVwtIQ0Q82MufrQ7aoovrdGXbTGPEcmA5QEVFRYfrmKXNj3C3QnTCAiLpwmTycUnfBH4KvNJpHhGre/DZNcDUrPkpQG0n69RIKgNGAvu6uK1ZQTh49Bjb9hzm7fM7PIg2y1snOwL523bzFVnTAVzRg89eCcyWNBPYSaZT/P3t1lkBLAWeAt4FPBYRIWkF8CNJf0emE3028F89yGKWmg2+A90K1MnGA7k8Vx+c9GncADwElAK3R8R6STcDlRGxgkzfyw8kbSFz5HFdsu16Sf8ObABagE/4CiwrVFVJAZnjU1hWYLo6Jno58E5gRvY2EXFzTz48Ih4AHmjX9qWs6aPAuzvZ9mvA13ry+Wb5YP3OBk4bXs5pwwelHcXslHS1E/0+oIHMVVjdvnHQzF6rqrbBp6+sIHW1gEyJiMU5TWJWhI40t7Jl9yEWz5mYdhSzU9alGwmBJzt7qKKZdd/Glw7QFjDHRyBWgLp6BPJ7wAclvUDmFJaAiIjzc5bMrAgcf4T7XBcQK0BdLSBX5zSFWZFaW9PAmKEDmTTSHehWeLr6MMUXASSdRuZx7mbWC9btbGDu5JFkHvFmVli6+jDFayVtBl4AfgVsB3r70SZmReVIcyubdx/i/Ck+fWWFqaud6F8FLgF+FxEzgSuB3+YslVkR2LDrAK1t4f4PK1hdLSDHImIvUCKpJCIeB+blMJdZv3f8Ee5zfQRiBaqrnej1koYBvwbulLQbOJa7WGb939qaBsYNK2fiCHcrWmHqagF5DmgkMybIH5J5Ku6wXIUyKwbrdtZz/hR3oFvh6moBuTwi2oA2khECJa3NWSqzfq6xuYUtuw9x9XmT0o5i1m0nGw/k48D/BM5oVzCG4050s27bUJu5A90d6FbITnYE8iMyl+t+Hbgxq/1gROzLWSqzfm5tjTvQrfCdbDyQBjJP4X1f38QxKw7rdjYwYUQ5E9yBbgWsq5fxmlkvytyBPirtGGY9kkoBkTRG0sOSNifvoztYZ56kpyStl7RW0nuzlv2bpBckrUlevifFCsahpha21h1y/4cVvLSOQG4EHo2I2cCjvLp/5bhG4I8iYg6wGPh7Sdl/sv15RMxLXmtyH9msd6zf2UAEfoSJFby0CsgSksuBk/e3tV8hIn4XEZuT6VpgNzC+zxKa5ci65A50j0JohS6tAjIhInYBJO+nnWhlSQuAgcDWrOavJae2bk3GbO9s22WSKiVV1tXV9UZ2sx5Zt7OB00cOYvzwTr+2ZgUhZwVE0iOSqjp4LTnF/UwCfgB8KLmZEeAm4GzgYmAM8PnOto+I5RFREREV48f7AMbSt66mwZfvWr/Q1TvRT1lEXNXZMkkvS5oUEbuSArG7k/VGAPcD/ysins7a965ksknSvwKf68XoZjlz4Ogxtu05zDsunJx2FLMeS+sU1gpgaTK9FLiv/QqSBgI/A74fEfe0WzYpeReZ/pOqnKY16yXrdx4AYO4UX8JrhS+tAnILsCgZpGpRMo+kCkm3Jeu8B3gjmbHY21+ue6ekdcA6YBzwV30b36x71lTXA3C+O9CtH8jZKawTScYWubKD9krgo8n0D4EfdrL9FTkNaJYjz+7Yz6xxQxk9dGDaUcx6zHeim/WRiODZ6nrmTfPpK+sfXEDM+sjO+iPUHWxi/lQXEOsfXEDM+sizOzL9H/OnvebJPWYFyQXErI88u6OeQQNKOGvi8LSjmPUKFxCzPvJs9X7OnzyKAaX+sbP+wd9ksz7Q1NLK+p0HmO8OdOtHXEDM+sD62gM0t7Yxzx3o1o+4gJj1gcrtmRGgL5rhDnTrP1xAzPrAyu37mTF2CKcN9xC21n+4gJjlWFtbULl9HxUzxqQdxaxXuYCY5di2PYfY33iMBS4g1s+4gJjl2Mrt+wGocP+H9TMuIGY5tvKFfYwbNpCZ44amHcWsV7mAmOXYyhf3UTF9DJnha8z6DxcQsxx6qeEo1fuO+PSV9UupFBBJYyQ9LGlz8t7hT5ek1qzBpFZktc+U9Eyy/Y+T0QvN8s5T2/YAcMmssSknMet9aR2B3Ag8GhGzgUeT+Y4ciYh5yevarPZvALcm2+8HPpLbuGbd8+SWvYwcPIBzJ41IO4pZr0urgCwB7kim7yAzrnmXJOOgXwHc253tzfpKRPDk1r1cOmssJSXu/7D+J60CMiEidgEk76d1st4gSZWSnpZ0vEiMBeojoiWZrwEmd/ZBkpYl+6isq6vrrfxmJ1W97wg764+w8HU+fWX9U87GRJf0CDCxg0VfPIXdTIuIWkmzgMckrQMOdLBedLaDiFgOLAeoqKjodD2z3vbbrZn+j4VnuIBY/5SzAhIRV3W2TNLLkiZFxC5Jk4DdneyjNnnfJukJYD7wE2CUpLLkKGQKUNvr/wCzHnpy615OG17OGeOHpR3FLCfSOoW1AliaTC8F7mu/gqTRksqT6XHAZcCGiAjgceBdJ9reLE0RwVNb97DwjLG+/8P6rbQKyC3AIkmbgUXJPJIqJN2WrHMOUCnpOTIF45aI2JAs+zzwp5K2kOkT+V6fpjc7iY27DrLnUDMLXzcu7ShmOZOzU1gnEhF7gSs7aK8EPppMPwnM7WT7bcCCXGY064nHN2XOyr75zPEpJzHLHd+JbpYDT2zazZzTR3DaCI//Yf2XC4hZL2toPMbqHfVcflZnV6eb9Q8uIGa97D+31NHaFlx+tk9fWf/mAmLWy57YVMfIwQOYN9UPULT+zQXErBe1tQVPbKrjjWeOp9SPL7F+zgXErBet3rGfPYeauOoc939Y/+cCYtaLHqx6iYGlJVxxtguI9X8uIGa9JCL4RdVLvGH2OIYPGpB2HLOccwEx6yVraxrYWX+Exed19AxRs/7HBcSslzxY9RJlJWLRuRPSjmLWJ1xAzHpBRPBg1S4uPWMso4Z4hGUrDi4gZr1g9Y79vLi3kWsvOD3tKGZ9xgXErBfcu2ongweUcvXcSWlHMeszLiBmPXT0WCs/X1vL4vMmMqw8lQdcm6XCBcSshx7e8DIHj7bwzgunpB3FrE+5gJj10L2rapg0chCXeuxzKzKpFBBJYyQ9LGlz8v6ap85JulzSmqzXUUlvS5b9m6QXspbN6/t/hRls33OYX2+u490VU/3sKys6aR2B3Ag8GhGzgUeT+VeJiMcjYl5EzAOuABqBX2at8ufHl0fEmj5JbdbO9596kVKJD7x+WtpRzPpcWgVkCXBHMn0H8LaTrP8u4MGIaMxpKrNTcLiphXsqq7lm7iSPPGhFKa0CMiEidgEk7yd78tx1wF3t2r4maa2kWyWVd7ahpGWSKiVV1tXV9Sy1WZafrq7hYFMLSxfOSDuKWSpyVkAkPSKpqoPXklPczyRgLvBQVvNNwNnAxcAY4POdbR8RyyOiIiIqxo/3CHHWO1pa27jtNy9w/pSRXDhtVNpxzFKRs4vWI+KqzpZJelnSpIjYlRSI3SfY1XuAn0XEsax970ommyT9K/C5Xglt1kX/saaWF/c2svz6i5DceW7FKa1TWCuApcn0UuC+E6z7PtqdvkqKDsr85L4NqMpBRrMOtbS28Y+PbebcSSP84EQramkVkFuARZI2A4uSeSRVSLrt+EqSZgBTgV+12/5OSeuAdcA44K/6ILMZAPclRx+fuWq2jz6sqKXy3IWI2Atc2UF7JfDRrPntwOQO1rsil/nMOtPY3MLf/nIT50320YeZ70Q3OwXfeWIrtQ1H+fL/mOOjDyt6LiBmXVS9r5Hv/nob115wOhfPGJN2HLPUuYCYdUFE8MX/qKJE4qZrzk47jllecAEx64IfPrODX/+uji9cczaTRg5OO45ZXnABMTuJbXWH+Ov7N/KG2eP4wCXT045jljdcQMxO4ODRY3zsB6soH1DCN991gTvOzbJ4+DSzTrS1BZ/98Rq27TnMDz68gIkj/cBEs2w+AjHrQETw5RXreWTjbr70B+ey8HXj0o5klndcQMzaiQhu/vkGfvD0i3zsjbP4o0vd72HWEZ/CMsvS3NLGF362jntX1fDhy2Zy49Vnu9/DrBMuIGaJuoNNfPKu1Ty9bR+funI2n/WzrsxOyAXEDHh048v8xb1rOdTUwq3vvYC3z5+SdiSzvOcCYkWtZn8jX7t/Iw9WvcTZE4dz17JLOHPC8LRjmRUEFxArSjvrj/CdJ7by48pqSgR/tuhMlr1pFuVlpWlHMysYLiBWNI40t/LrzXXcU1nDY8+/TGmJeNdFU7jhitlMHuXHk5idKhcQ67cigup9R/jNlj08uvFlfrNlD00tbYwbNpCPvekMPnDJdBcOsx5IpYBIejfwFeAcYEEykFRH6y0GvgWUArdFxPGRC2cCdwNjgNXA9RHR3AfRLQ9FBPsbj7F972F27G1k8+6DrK1pYN3OBuobjwEwZfRg3rdgGovOncCCmWMYUOpboMx6Kq0jkCrgHcB3O1tBUinwbTJD3tYAKyWtiIgNwDeAWyPibknfAT4C/HPuY1tviAha2oKW1uBYWxstrUFLaxvH2pL31uDosVYONbVw6GgLh5tb/nu6qYWDTS3sOdRM3cGj1B1sYveBJg42tbyy/9IScdaE4SyeM5Hzp4zioumjOXPCMF+Sa9bL0hrSdiNwsh/oBcCWiNiWrHs3sETSRuAK4P3JeneQOZrJWQH54s/W8cwL+4DML7/jInul6HDyVeu/dll2e3Tc/urNO9xvZ/s84X473aYr63fx39QuY0tr0NKWKRDdJcGwgWWMHTaQ8cPLOWvicH7vdeOYNnYo08cMYfrYIUwdM4RBA9wZbpZr+dwHMhmozpqvAV4PjAXqI6Ilq/0146YfJ2kZsAxg2rRp3Qpy+qjBnJV9aac6nHxVQXx1e7tMXdjm1Z+RtU6nn93x+q9Z1smHnOp+u/pvyjagVJSVljCgJPNeVioGlpZQlswPKBVlJZn2wQNKGVZextDyMoYNKmNYeeY1eEApJSU+kjDLBzkrIJIeASZ2sOiLEXFfV3bRQVucoL1DEbEcWA5QUVHRrT99P3H567qzmZlZv5azAhIRV/VwFzXA1Kz5KUAtsAcYJaksOQo53m5mZn0ony9FWQnMljRT0kDgOmBFZE6sPw68K1lvKdCVIxozM+tFqRQQSW+XVANcCtwv6aGk/XRJDwAkRxc3AA8BG4F/j4j1yS4+D/yppC1k+kS+19f/BjOzYqf2V9T0ZxUVFVFZ2eEtJ2Zm1glJqyKion17Pp/CMjOzPOYCYmZm3eICYmZm3eICYmZm3VJUneiS6oAXu7n5ODL3oOSbfM0F+ZvNuU5NvuaC/M3W33JNj4jx7RuLqoD0hKTKjq5CSFu+5oL8zeZcpyZfc0H+ZiuWXD6FZWZm3eICYmZm3eIC0nXL0w7QiXzNBfmbzblOTb7mgvzNVhS53AdiZmbd4iMQMzPrFhcQMzPrFheQUyBpnqSnJa2RVClpQdqZjpP0SUmbJK2X9Ddp58km6XOSQtK4tLMcJ+mbkp6XtFbSzySNSh/ieb8AAASESURBVDnP4uT/3xZJN6aZ5ThJUyU9Lmlj8r36dNqZskkqlfSspJ+nnSWbpFGS7k2+XxslXZp2JgBJn03+P1ZJukvSoJ7u0wXk1PwN8L8jYh7wpWQ+dZIuB5YA50fEHOD/pBzpFZKmAouAHWlnaedh4LyIOB/4HXBTWkEklQLfBq4GzgXeJ+nctPJkaQH+LCLOAS4BPpEnuY77NJmhHvLNt4BfRMTZwAXkQUZJk4FPARURcR5QSmaMpR5xATk1AYxIpkeSPyMhfhy4JSKaACJid8p5st0K/AUnGHY4DRHxy2TMGYCnyYxsmZYFwJaI2BYRzcDdZP4gSFVE7IqI1cn0QTK/CCenmypD0hTg94Hb0s6STdII4I0kYxRFRHNE1Keb6hVlwGBJZcAQeuH3lwvIqfkM8E1J1WT+yk/tr9Z2zgTeIOkZSb+SdHHagQAkXQvsjIjn0s5yEh8GHkzx8ycD1VnzNeTJL+rjJM0A5gPPpJvkFX9P5g+TtrSDtDMLqAP+NTm9dpukoWmHioidZH5n7QB2AQ0R8cue7jdnY6IXKkmPABM7WPRF4ErgsxHxE0nvIfNXRk/Hfu+NXGXAaDKnGS4G/l3SrOiDa7RPkusLwFtynaEzJ8oWEfcl63yRzKmaO/syWzvqoC1vjtgkDQN+AnwmIg7kQZ4/AHZHxCpJb047TztlwIXAJyPiGUnfAm4E/jLNUJJGkzmqnQnUA/dI+kBE/LAn+3UBaSciOi0Ikr5P5rwrwD304eHzSXJ9HPhpUjD+S1IbmYem1aWVS9JcMl/W5yRB5hTRakkLIuKlXOc6UbbjJC0F/gC4si+K7QnUAFOz5qeQJ6dHJQ0gUzzujIifpp0ncRlwraRrgEHACEk/jIgPpJwLMv8vayLi+JHavWQKSNquAl6IiDoAST8FFgI9KiA+hXVqaoE3JdNXAJtTzJLtP8jkQdKZwEBSfhJoRKyLiNMiYkZEzCDzg3VhXxWPk5G0GPg8cG1ENKYcZyUwW9JMSQPJdG6uSDkTylT+7wEbI+Lv0s5zXETcFBFTku/VdcBjeVI8SL7f1ZLOSpquBDakGOm4HcAlkoYk/1+vpBc6930Ecmr+GPhW0gl1FFiWcp7jbgdul1QFNANLU/6LuhD8E1AOPJwcIT0dEX+SRpCIaJF0A/AQmatjbo+I9Wlkaecy4HpgnaQ1SdsXIuKBFDMVgk8CdyZ/DGwDPpRyHpLTafcCq8mcsn2WXnisiR9lYmZm3eJTWGZm1i0uIGZm1i0uIGZm1i0uIGZm1i0uIGZm1i0uIGZm1i0uIGZm1i0uIGYpknRxMibJIElDk/Eazks7l1lX+EZCs5RJ+isyz3QaTOY5Sl9POZJZl7iAmKUseeTFSjKPx1kYEa0pRzLrEp/CMkvfGGAYMJzMkYhZQfARiFnKJK0gMwrhTGBSRNyQciSzLvHTeM1SJOmPgJaI+FEyNvqTkq6IiMfSzmZ2Mj4CMTOzbnEfiJmZdYsLiJmZdYsLiJmZdYsLiJmZdYsLiJmZdYsLiJmZdYsLiJmZdcv/B9Rr/k/wL3v2AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "y = x.tanh()\n",
    "xyplot(x, y, 'tanh')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_cc8mnr5",
    "id": "A3BD6777A0254AB7A99405AFDB9D8131",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "依据链式法则，tanh函数的导数\n",
    "\n",
    "\n",
    "$$\n",
    "\\text{tanh}'(x) = 1 - \\text{tanh}^2(x).\n",
    "$$\n",
    "\n",
    "\n",
    "下面绘制了tanh函数的导数。当输入为0时，tanh函数的导数达到最大值1；当输入越偏离0时，tanh函数的导数越接近0。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "graffitiCellId": "id_i1mi17x",
    "id": "CB16F4B33E664E14BCE8E52D8B37C47F",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXxcd3nv8c8zWqzN2ix5lWx5SxzHcTYnLFlYwhKSNGkLpQkQoAu5KQ1QltJQaEqh90LhdqEQejEQdpKSsLnUIRQITSgkWI4TO7ZjrDi2JW+SJVu29mWe+8fMOGNFy0iaM2ek+b5fLyHNmaMzT7Ck7/yW8/uZuyMiIrkrEnYBIiISLgWBiEiOUxCIiOQ4BYGISI5TEIiI5Lj8sAuYrJqaGm9oaAi7DBGRGWXr1q3H3b12tOdmXBA0NDTQ2NgYdhkiIjOKmR0Y6zl1DYmI5DgFgYhIjlMQiIjkOAWBiEiOUxCIiOS4wILAzO4xs1Yze3qM583M/tXMmsxsu5ldElQtIiIytiBbBF8Frh3n+dcBq+MftwH/FmAtIiIyhsCCwN0fATrGOeUm4Ose8xhQaWaLgqpHJGgnugf41uMH6BscDrsUkUkJc4xgCdCc9LglfuwFzOw2M2s0s8a2traMFCcyGZ09g7z5S4/z4e8/zZ99cyv9QwoDmTnCDAIb5diou+S4+0Z33+DuG2prR71DWiQ0vQPDvPWex2lq7eKtL1nGw3vaeNe3txGNatMnmRnCXGKiBahPelwHHA6pFpEp+9H2wzzV0sndb7qE69cvYmFFEZ/68R62HjzBZQ3VYZcnMqEwWwSbgLfGZw+9GOh09yMh1iMyJf+54whLKou57oKFALz1JQ0U5kf4z+36cZaZIcjpo/cCvwbONbMWM/sTM7vdzG6Pn7IZ2Ac0AV8E3hlULSJB6ewZ5H+ajnP9+kWYxXo7y+bk8/Jzannw6SPqHpIZIbCuIXe/ZYLnHfjzoF5fJBN+susog8POdRecPeHt+vWL+MmuY+oekhlBdxaLTEOiW+jCuoqzjl9z3gJ1D8mMoSAQmaLO3hd2CyUkuoc27zhCrPErkr0UBCJTtO3gCQaHnZefO/qU5lesmU/r6X72t/dkuDKRyVEQiEzRU82dmMH6uspRn7+ovjJ+3slMliUyaQoCkSl6quUkq+eXUTZn9DkXq+eXUVyQx5MKAslyCgKRKXB3nmo+yYVjtAYA8vMiXLCkgqdaFASS3RQEIlPQcqKX9u4BLqwfOwgALqyvYOfhUwwMRTNUmcjkKQhEpiDxLv+iCYOgkoGhKHuOns5EWSJToiAQmYKnmk9SmB/h3IVzxz0v0XX0pLqHJIspCESm4KnmTtYtLqcgb/xfobqqYuaVFmrmkGQ1BYHIJA0NR9lxqHPC8QEAM+PC+krNHJKspiAQmaSDHT30Dg5z/uKKiU8G1i0uZ19blzarkaylIBCZpL2tXUDsPoFUrJxfRtThuePdQZYlMmUKApFJaooHwcoUg2D1/NiA8t5jXYHVJDIdCgKRSWpq7WJJZfGYdxSPtKK2lIg9HyAi2UZBIDJJe1tPp9waACgqyKO+ukRBIFlLQSAyCdGo09TalfL4QMLq+WXsbdVNZZKdFAQik3DoZC99g9FJB8Gq+XN57ng3Q8NaakKyj4JAZBIS3TurJh0EZQwOOwc6tDeBZB8FgcgkJLp3JhsEiRaEZg5JNlIQiExCU2sXtXPnUFlSOKnvSwwuP9umIJDsoyAQmYS9rV2sqp1cawBiexgvrihi7zENGEv2URCIpMg9NmNost1CCasWzD1zV7JINlEQiKSoo3uA031DrKgtndL3r6gp5UB7D+6e5spEpkdBIJKi/e2xGT8N86YWBMvmldDVP0R790A6yxKZNgWBSIoOtMcWjVs2r2RK358IkMR1RLKFgkAkRfvbe4gY1FVNLQgSAbL/uO4lkOyiIBBJ0YH2bhZXFlOYP7Vfm7qqEiKmFoFkHwWBSIr2t/dMeXwAoDA/wpKqYt1dLFlHQSCSogPt3VMeH0hYVl16ZtBZJFsoCERS0NkzyMmewWm1CCA2TqCuIck2gQaBmV1rZnvMrMnM7hzl+aVm9rCZbTOz7WZ2XZD1iEzVgY7YH++l02wRNMwr5WTPICd7NIVUskdgQWBmecDdwOuAtcAtZrZ2xGkfAb7j7hcDNwOfD6oekemY7j0ECYmupQPqHpIsEmSL4HKgyd33ufsAcB9w04hzHCiPf10BHA6wHpEpOxDfeH5p9TRbBDXxewk0YCxZJMggWAI0Jz1uiR9L9lHgLWbWAmwG3jXahczsNjNrNLPGtra2IGoVGdf+9h4WlhdRXJg3reskgiQRLCLZIMggsFGOjVxk5Rbgq+5eB1wHfMPMXlCTu2909w3uvqG2tjaAUkXGl44ZQxDbv3hheZFmDklWCTIIWoD6pMd1vLDr50+A7wC4+6+BIqAmwJpEpuRAR09aggA0c0iyT5BBsAVYbWbLzayQ2GDwphHnHASuATCz84gFgfp+JKv0DQ7Tdrqf+ikuLTFSfXUJzSfUIpDsEVgQuPsQcAfwELCb2OygnWb2MTO7MX7a+4F3mNlTwL3A211r9EqWaYn/0a6f5kBxQn1VCcdO9dM3OJyW64lMV36QF3f3zcQGgZOP3ZX09S7giiBrEJmu5hO9ANRVFaflevXVsescOtnLyinsdiaSbrqzWGQCLR3pbREkZg41awqpZAkFgcgEmk/0UpgfobZsTlqulwiUREtDJGwKApEJNHf0UFdZTCQy2ozoyastm0NhfuRMS0MkbAoCkQm0nOilLk3dQgCRiFFXVayZQ5I1FAQiE2g+0UN9mgaKE+qrSjioFoFkCQWByDhO98WWn57q9pRjqa8uprlDYwSSHRQEIuNoiQ/oJqZ8pkt9VQmdvYOc6htM63VFpkJBIDKOxBTPdN1VnFCvKaSSRRQEIuNoPtMiSHMQVCWCQN1DEj4Fgcg4Wk70UFKYR1VJQVqvm+hqatHMIckCCgKRcTR39FJfVYJZeu4hSKgoLmBuUb66hiQrKAhExtFyoiftA8UAZkZ9VYnuLpasoCAQGYO7x24mS/NAcUJsCqlaBBI+BYHIGE72DNLVP5S2VUdHirUIetDK6xI2BYHIGJrTvA/BSPXVJfQNRmnr6g/k+iKpUhCIjKElzfsQjJQYe9AUUgmbgkBkDM1p3odgpMS9BJpCKmFTEIiMoflEDxXFBZQXpfcegoS6Kt1dLNlhwq0qzWw+se0kFwO9wNNAo7tHA65NJFSxGUPBdAsBFBfmUVM2R11DEroxg8DMXgHcCVQD24BWoAj4XWClmT0A/KO7n8pEoSKZ1tzRw+r5cwN9jfpq7Usg4RuvRXAd8A53PzjyCTPLB24AXg18N6DaREKTuIfglWvmB/o6S6tLeOLgiUBfQ2QiYwaBu//lOM8NAT8IpCKRLNDW1U//UDSwm8kS6qtK+NH2IwwNR8nP05CdhGPCnzwz+4aZVSQ9bjCznwVblki4Ev32QSwvkay+upjhqHOksy/Q1xEZTypvQX4JPG5m15nZO4CfAP8SbFki4UpM6Uz3PgQj1WvmkGSBCWcNufsXzGwn8DBwHLjY3Y8GXplIiBI3ky0JcNYQJG1QowFjCVEqXUO3AvcAbwW+Cmw2swsDrkskVM0dPdSUFVJSOOF7pWlZVFFEXsQ0hVRClcpP+euBK929FbjXzL5PLBAuDrIwkTA1n+gJfKAYID8vwqKKIrUIJFQTtgjc/XfjIZB4/BvgRYFWJRKy5o7ewJaWGKm+qkRjBBKqMYPAzD5iZtWjPefuA2b2SjO7IbjSRMIxNBzl0MlelgY8YyghdlOZuoYkPON1De0A/sPM+oAngDZidxavBi4Cfgr8n8ArFMmwI519DEedpRlsEbSd7qdvcJiigryMvKZIsjFbBO7+Q3e/Argd2AnkAaeAbwKXu/t73b1tvIub2bVmtsfMmszszjHOeaOZ7TKznWb27an/p4ikx5lVRzMwRgCwdJ5WIZVwpTJ9dC+wd7IXNrM84G5iy1C0AFvMbJO770o6ZzXwIeAKdz8RX+BOJFQHA15+eqTnVyHtZVXAaxuJjCaV1UfPAT4ANCSf7+6vnOBbLwea3H1f/Dr3ATcBu5LOeQdwt7ufiF+z9QVXEcmw5hM95EWMRRVFGXm9MxvUqEUgIUll+uj9wP8DvgQMT+LaS4DmpMctvHC20TkAZvY/xLqePuruPx55ITO7DbgNYOnSpZMoQWTyDnb0sqSyOGNr/9SWzaGoIMLBdgWBhCOVIBhy93+bwrVtlGMjd+nOJzb4/HKgDnjUzNa5+8mzvsl9I7ARYMOGDdrpWwLV3NET+BpDycyMuvhG9iJhGG/6aHV8+uh/mNk7zWxR4thY00pHaAHqkx7XAYdHOeeH7j7o7s8Be4gFg0homjt6MjZjKKG+qlh3F0toxmsRbCX2Dj7xzj55WWoHVkxw7S3AajNbDhwCbgbeNOKcHwC3AF81sxpiXUX7UitdJP26+4do7x7IyF3FyeqrS2g8oH0JJBzj7UewfDoXdvchM7sDeIhY//897r7TzD5GbKvLTfHnXmNmu4iNP/ylu7dP53VFpiPRPZP5FkEJp/uG6OwZpKIkmD2SRcaS0opaZvZSXjhr6OsTfZ+7bwY2jzh2V9LXDrwv/iESukT3TMaDIGnmUEVJxQRni6RXKtNHvwGsBJ7k+VlDDkwYBCIzTabvIUioS9qXYN0SBYFkViotgg3A2vi7d5FZrbmjh7I5+VRluHsmcXexZg5JGFKZKP00sDDoQkSyQXNHD3VVxZiNNvs5OOVFBVQUF2jmkIQilRZBDbDLzH4D9CcOuvuNgVUlEpKDHT0srykN5bXrq4vPdE2JZFIqQfDRoIsQyQbuTvOJHq4+pzaU16+vKmHPsdOhvLbktlQWnfvvTBQiEra2rn76BqMZnzGUUF9dws+eaSUadSKRzHZNSW5LZc/iF5vZFjPrMrMBMxs2s1OZKE4kk8KaOppQX1XMwFCUtq7+iU8WSaNUBos/R+zu371AMfCn8WMis8qZfQgyuM5Qsrrq56eQimRSSssrunsTkOfuw+7+FWKLxInMKok/wJleXiIhsRGOppBKpqUyWNxjZoXAk2b2KeAIEM60CpEAHezoYf7cOaFtF1lXFb+7WFNIJcNSaRHcGj/vDqCb2Iqivx9kUSJhaD6R+VVHkxUV5LGgfI66hiTjUgmC33X3Pnc/5e5/5+7vA24IujCRTGvu6M340hIj1WtfAglBKkHwtlGOvT3NdYiEamAoypHOLAiC6hJ1DUnGjTlGYGa3ENs/YLmZbUp6ai6gpaJlVjl8speohzd1NKG+qpgfPtnL4HCUggxtlSky3mDxr4gNDNcA/5h0/DSwPciiRDLtzKqjVeFMHU2oqy4h6rFgWjZPczIkM8bbmOYAcAB4SebKEQnHmQ1p5oXdIkjcS6AgkMxR21OEWIugMC/CgrlFodaRuJlNi89JJikIRID9x7upry4OfY2fRRXFFOZFONDeHWodklvGDAIz+1n88z9krhyRcOw/Ht7y08nyIsbSeSU8d1xBIJkz3mDxIjN7GXCjmd0HnPVWyd2fCLQykQyJRp397d1ctbom7FIAaJhXyn61CCSDxguCu4A7gTrgn0Y858ArgypKJJOOnuqjfyhKQxa0CACW15Tw6N42LUctGTPerKEHgAfM7G/c/eMZrEkko/bHu2GyoWsIoKGmlP6hKEdO9bGkMtzprJIbUtmY5uNmdiNwdfzQL9z9R8GWJZI5z8W7YbKmRRCfNrr/eLeCQDIilY1pPgG8B9gV/3hP/JjIrLD/eDdz8iMsKg936mhCIpA0YCyZksoy1NcDF7l7FMDMvgZsAz4UZGEimfLc8R6WzSvJmv74heVFzMmPnOmyEglaqvcRVCZ9XRFEISJh2d/eTUMW3cUbiZhmDklGpdIi+ASwzcweJjaF9GrUGpBZYjjqHGzv4Zo188Mu5SwNNSU0tXaFXYbkiFQGi+81s18AlxELgr9y96NBFyaSCYdP9jIwnD1TRxMaakp5+Jk2hqNOXpZ0WcnslUqLAHc/Amya8ESRGSbR/ZJNXUMQmzk0MBzl8Mnw90iQ2U9rDUlOy7Z7CBISLZR9GjCWDFAQSE57tq2b0sLYXsHZZEVtPAjaNE4gwRtv0bnq8T5SubiZXWtme8ysyczuHOe8N5iZm9mGqfxHiExVU2sXK+eXYZZd/fC1ZXMoL8rXgLFkxHhjBFuJrSlkwFLgRPzrSuAgsHy8C5tZHnA38GqgBdhiZpvcfdeI8+YC7wYen+J/g8iUPdvWxUtWzAu7jBcwM1bNL+NZtQgkA8ZsEbj7cndfATwE/I6717j7POAG4HspXPtyoMnd97n7AHAfcNMo530c+BTQN+nqRaahq3+II519rJxfFnYpo1pZW0ZTq8YIJHipjBFc5u6bEw/c/UHgZSl83xKgOelxS/zYGWZ2MVA/0dpFZnabmTWaWWNbW1sKLy0ysWfj3S4ra7MzCFbNL+N4Vz+dPYNhlyKzXCpBcNzMPmJmDWa2zMw+DLSn8H2jdbr6mSfNIsA/A++f6ELuvtHdN7j7htra2hReWmRiif73VVnaIkjU1dR2OuRKZLZLJQhuAWqB7wM/AObHj02kBahPelwHHE56PBdYB/zCzPYDLwY2acBYMuXZti7yI8aykDesH0siCJ5V95AELJU7izuIrT46WVuA1Wa2HDgE3Ay8Kem6ncCZLaHidy9/wN0bp/BaIpPW1NpFQ00pBXnZOYu6rqqEwvwITRowloBNGARmVgt8EDgfOLNOr7uPu0OZuw+Z2R3EBpvzgHvcfaeZfQxodHfdqSyhamrr4pz5c8MuY0x5EWNFTammkErgUlli4lvAvxObLXQ78DYgpRHb+CDz5hHH7hrj3Jenck2RdBgYinKgvYfXrVsYdinjWjm/jB0tnWGXIbNcKm3iee7+ZWDQ3f/b3f+YWH++yIx1sKOb4ahn7UBxwqraMppP9NA3OBx2KTKLpRIEiblrR8zs+viUz7oAaxIJ3JkZQ7XZ2zUEsQFjd+1WJsFKpWvo782sgtg0z88C5cB7A61KJGB7jnZhBivnZ9dicyOduzAWVL89dprzFpWHXI3MVuO2COLLRKx29053f9rdX+Hul2qgV2a6Z46eomFeKSWFKa3EHprlNaUU5Bm7j+heAgnOuEHg7sPAjRmqRSRjnjl6mjULs7tbCKAgL8Kq+XN55uipsEuRWSyVMYJfmdnnzOwqM7sk8RF4ZSIB6RkYYn97N2sWzoyulvMWzuUZtQgkQKm0i18a//yxpGMOjHsfgUi22nusC/fn+9+z3ZpFc/netkOc7BmgsqQw7HJkFkrlzuJXZKIQkUxJdLOct2hmBMG58ZbLM0dP8+IsXDJbZr5U7ix+3yiHO4Gt7v5k+ksSCdbuI6cpKcyjvio71xga6bx4y+WZI6cUBBKIVMYINhC7o3hJ/OM24OXAF83sg8GVJhKMZ46e4pwFc4lEsmtXsrHUzp1DdWkhzxzVOIEEI6U7i4FL3P397v5+YsFQC1wNvD3A2kTSzt155ujpGdMtBLHdytYsnMtuBYEEJJUgWAoMJD0eBJa5ey/QH0hVIgE5dqqfkz2DM2bGUMKaheX89uhpolGf+GSRSUpl1tC3gcfM7Ifxx78D3GtmpcCusb9NJPvsjg8Uz5QZQwlrFs2ld3CY/e3drMjSHdVk5kpl1tDHzWwzcCWxXcduT9oz4M1BFieSbomVPM9fPLNaBOsWVwCw41CngkDSLqX76919K7A14FpEAre9pZMVtaXMLSoIu5RJWb2gjDn5EXa0dHLTRUsm/gaRScjOrZlEArLj0EnWL6kIu4xJK8iLsHZxOdsPaW8CST8FgeSMY6f6OHaqnwvqKsMuZUrWL6lg56FOhjVgLGmmIJCckRgfWF8381oEABfUVdI9MMxzx7V1paSXgkByxvZDnUQM1s7Qdf0TAbZdW1dKmikIJGfsaDnJqvlllM7J7j0IxrKytoySwjwFgaSdgkBygruz41An62fo+ABAXsRYt7iCHRowljRTEEhOONzZx/GugRk7PpBwQV0FOw93MjQcDbsUmUUUBJITth08AcCFM7hFAHBhfSV9g1EtQCdppSCQnNC4/wTFBXmsnWF3FI90WUMVAFv2d4RcicwmCgLJCVv2d3Dx0koK8mb2j/yiimKWVBbTuP9E2KXILDKzfytEUnC6b5DdR05xWUN12KWkxWUNVWzZ34G7biyT9FAQyKy37eBJos6sCYINDdW0nu6nuaM37FJkllAQyKzXuL+DvIhx0dKZPVCckAg0jRNIuigIZNbbsv8EaxeVUzZDbyQbafX8MsqL8mk8oCCQ9FAQyKw2OBxlW/MJNsRn28wGkYixoaGaLRowljQJNAjM7Foz22NmTWZ25yjPv8/MdpnZdjP7mZktC7IeyT1PNZ+kbzDK5bNkfCDh8uXVNLV20XqqL+xSZBYILAjMLA+4G3gdsBa4xczWjjhtG7DB3dcDDwCfCqoeyU2P7D1OxOClK2vCLiWtrlwV++/5ZdPxkCuR2SDIFsHlQJO773P3AeA+4KbkE9z9YXfviT98DKgLsB7JQY/ubePC+koqSmbWjmQTWbuonHmlhTy6V0Eg0xdkECwBmpMet8SPjeVPgAdHe8LMbjOzRjNrbGtrS2OJMpt19gzyVPNJrlpdG3YpaReJGFeuruHRvceJaqMamaYgg8BGOTbqT6yZvQXYAHx6tOfdfaO7b3D3DbW1s++XWoLxq2ePE3W4evXs6hZKuGp1Lce7+rXukExbkEHQAtQnPa4DDo88ycxeBXwYuNHd+wOsR3LMI3uPM3dOPhfWz477B0a6Kh5wj+5VK1mmJ8gg2AKsNrPlZlYI3AxsSj7BzC4GvkAsBFoDrEVyjLvz6N42XrJy3oxfX2gsC8qLOHfBXB5REMg0BfYb4u5DwB3AQ8Bu4DvuvtPMPmZmN8ZP+zRQBtxvZk+a2aYxLicyKXtbu2g50cvV58zursSXnVvLludOcKpvMOxSZAYL9FZLd98MbB5x7K6kr18V5OtL7vrx00cxg9esXRB2KYF67fkL2PjIPh5+ppWbLhpvLobI2GZnm1ly3o+fPsqlS6uYX14UdimBuri+itq5c/jx00fDLkVmMAWBzDoH23vYdeQU165bGHYpgYtEjNeev4Bf7Gmjd2A47HJkhlIQyKzz0M7Yu+PXnj/7gwDg2vMX0Ts4rEFjmTIFgcw6Dz59hHVLyqmvLgm7lIx40YpqKksK1D0kU6YgkFnlYHsPTxw8yevWLQq7lIwpyIvwmrUL+MnOo/QMDIVdjsxACgKZVR7Y2kzE4Pcvya0ZNG+4tJ7ugWE271CrQCZPQSCzxnDUeWBrC1eurmVRRXHY5WTUZQ1VNMwr4f7G5olPFhlBQSCzxq+ePc7hzj7+4NLcW8TWzHjDpXU8/lwHB9t7Jv4GkSQKApk17m9soaK4gFfP8pvIxvL6S+swg/u3qlUgk6MgkFmh9VQfDz59hN+7eAlFBXlhlxOKRRXFvOycWu79TTP9Q7qnQFKnIJBZ4Wu/3s9Q1PmjKxrCLiVUf3rlCo539fPDbS9Y6FdkTAoCmfF6Bob45mMHee3ahSybVxp2OaG6YtU81iycyxcf3Ye7NqyR1CgIZMa7v7GFzt5B3nH1irBLCZ2ZcdvVK9jb2sUvfqs7jSU1CgKZ0foGh9n4yD4uWVrJpcuqwi4nK9ywfjELy4v47M/2qlUgKVEQyIz2jV8f4NDJXj7wmnPDLiVrFOZHePc1q3ni4En+a9exsMuRGUBBIDNWZ+8gn3u4iavPqeWlq2bnvsRT9cYNdayoLeVTD+1haDgadjmS5RQEMmN9/uEmTvUNcue1a8IuJevk50X44GvX0NTaxXcaW8IuR7KcgkBmpJ2HO/nyL5/j9ZfUsXZxedjlZKXXnr+Ayxuq+eSDu2k93Rd2OZLFFAQy4wwNR/mr726nsqSAj1x/XtjlZC0z4xOvv4C+oSh/+8OdYZcjWUxBIDPOFx7Zx9OHTvF3N66jsqQw7HKy2sraMt5zzWoefPooP9qum8xkdAoCmVF+/Ww7//iTPVy/fhHXXZAbO5BN121Xr+Ci+kru/O4Onm3rCrscyUIKApkxjnb28a57n2B5TSn/8Pr1mFnYJc0IBXkRPv/mSyjMj3D7N7bS3a/Na+RsCgKZEU72DPD2r/yGnoFhvnDrpZTNyQ+7pBllcWUxn73lYp5t6+L2b27VonRyFgWBZL2u/iHe/pUt7GvrZuOtG1g1f27YJc1IV6yq4ZOvX8+je4/z7nu36f4COUNBIFmt9XQft2x8jB2HOvnsmy7mytW6cWw63rihnrtuWMtDO49xm7qJJE5BIFlr95FT/P7nf0VTaxcbb72U156vweF0+OMrl/O/f28dv9jTys0bH+PQyd6wS5KQKQgk60Sjztd+tZ+b7v4f+oei/Pv/ejHXnJebu44F5c0vWsYX37qBfW1dvO5fHmHzjiNhlyQhUhBIVtl1+BRv/MKv+dtNO7lyVQ0/fs9VrK+rDLusWema8xbwn+++iuU1pbzzW0/wp19rpLlD+x3nIptpy9Ru2LDBGxsbwy5D0uy3x07zuZ838R/bD1NVUsid167hDzbUaYpoBgwMRfnyL5/jX3+2l+Go84eX1XP7y1eypLI47NIkjcxsq7tvGPU5BYGE5VTfIA8/08p9v2nm1/vaKSnM49aXLOOdL1tFRUlB2OXlnMMne/nsz/dyf2MLDlyzZj5v3FDPlatrcnYf6NlEQSBZIRp19rd388hv2/jp7lYe29fOUNRZUlnMLZfX86YXLaO6VEtGhK3lRA/ffOwg9zc20949QHFBHleuruFV583nilU1LKksVkttBgotCMzsWuAzQB7wJXf/5Ijn5wBfBy4F2oE/dPf9411TQZD93J327gEOtHdzoL2HPcdOs6Olkx2HOjndF5uuuLK2lFetXcCrzlvAJUuryIvoD0u2GRiK8ti+dn66+xg/3XWMw52xFUznlRZyQV0F6+sqWTW/jGXVJTTMK1UrLsuFEgRmlgf8Fng10AJsAW5x911J57wTWO/ut5vZzUcFTKQAAAh/SURBVMDvufsfjnddBUH6uTvDUWco6gwORxkajn0ejDpDw1EGh53egWG6+ofo7h+iK/7RHf841TfE8a5+2k7309bVz7HOProHnr9ztSDPWLOwnPV1Fayvq+Dy5fNYXpPbm8zPNO7O7iOn2Xqgg6daOtnR0sne1tNEk/58lBflM7+8iOrSQmrKCplXOofq0kKqSwspLsyjJP5RXJAf+1yYR3FBHnMKIhREIuTlGfkRIy9i5EciRAy1PNJovCAI8j79y4Emd98XL+I+4CZgV9I5NwEfjX/9APA5MzMPIJ2+s6WZjY/uAzizj+tZL+KjfjnquX7Wuf7CY2NUP6VrjXEuE547/mtF3RkadoaisT/0U2UGZYX51MydQ23ZHNYsnMvVq2tZNq+EZfNKWFpdSn11MXPy1cc8k5kZaxeXs3ZxObfGj/UODHOgI9bqO9jew8GOHo539dPePcCeo6fp6G7nRM/gtF73+WCIf86LYMR+7sDitXHmmGEkssN4PkjOHEs656zn4/+T7thJd5C955rV/M6Fi9N6TQg2CJYAzUmPW4AXjXWOuw+ZWScwDziefJKZ3QbcBrB06dIpFVNVWsi5C5KWJrCzPiVeZ+TT8eMpnpt0giU9sLOOj3bsheee/fMzzWuNUXdBXuwXqyD+C5afZxREYp+TjxfkGcUFeZQV5VM2J5/SObHPZXPyKS7II6JunZxUXJjHmoXlrFk49sZAQ8NROnsH6RkYpndwmJ6BYXoGhugdiH3dOzBM/3CU4eEoQ9HnW6bPf44fH449HopGcX/+DU7szY3HjvnZb6acpGPxb4gd8zPPjTyWVgF0tlQUB9P9FmQQjPbXYeT/Namcg7tvBDZCrGtoKsW8eu0CXr1WNyWJZFJ+XoR5ZXOYF3YhMq4gbyhrAeqTHtcBI3fGOHOOmeUDFUBHgDWJiMgIQQbBFmC1mS03s0LgZmDTiHM2AW+Lf/0G4OdBjA+IiMjYAusaivf53wE8RGz66D3uvtPMPgY0uvsm4MvAN8ysiVhL4Oag6hERkdEFuruHu28GNo84dlfS133AHwRZg4iIjE+LzomI5DgFgYhIjlMQiIjkOAWBiEiOm3Grj5pZG3Bgit9ew4i7lrNIttamuiYnW+uC7K1NdU3eVGpb5u61oz0x44JgOsyscaxFl8KWrbWprsnJ1roge2tTXZOX7trUNSQikuMUBCIiOS7XgmBj2AWMI1trU12Tk611QfbWpromL6215dQYgYiIvFCutQhERGQEBYGISI7LuSAws4vM7DEze9LMGs3s8rBrSjCzd5nZHjPbaWafCruekczsA2bmZlYTdi0AZvZpM3vGzLab2ffNrDLkeq6N//s1mdmdYdaSYGb1Zvawme2O/1y9J+yakplZnpltM7MfhV1LMjOrNLMH4j9fu83sJWHXBGBm743/Oz5tZveaWVE6rptzQQB8Cvg7d78IuCv+OHRm9gpiezivd/fzgf8bcklnMbN64NXAwbBrSfJfwDp3Xw/8FvhQWIWYWR5wN/A6YC1wi5mtDaueJEPA+939PODFwJ9nSV0J7wF2h13EKD4D/Njd1wAXkgU1mtkS4N3ABndfR2x5/7Qs3Z+LQeBAYpPVCl64a1pY/gz4pLv3A7h7a8j1jPTPwAcJZCfWqXH3n7j7UPzhY8R2wQvL5UCTu+9z9wHgPmLBHip3P+LuT8S/Pk3sD9qScKuKMbM64HrgS2HXkszMyoGrie2XgrsPuPvJcKs6Ix8oju/oWEKa/n7lYhD8BfBpM2sm9q47tHeRI5wDXGVmj5vZf5vZZWEXlGBmNwKH3P2psGsZxx8DD4b4+kuA5qTHLWTJH9wEM2sALgYeD7eSM/6F2JuLaNiFjLACaAO+Eu+2+pKZlYZdlLsfIvY36yBwBOh095+k49qBbkwTFjP7KbBwlKc+DFwDvNfdv2tmbySW+q/KgrrygSpizffLgO+Y2YpMbd05QW1/DbwmE3WMNF5d7v7D+DkfJtYF8q1M1jaCjXIsa1pPZlYGfBf4C3c/lQX13AC0uvtWM3t52PWMkA9cArzL3R83s88AdwJ/E2ZRZlZFrJW5HDgJ3G9mb3H3b0732rMyCNx9zD/sZvZ1Yv2SAPeTwWbpBHX9GfC9+B/+35hZlNjCUm1h1mZmFxD7wXvKzCDW/fKEmV3u7kfDqiupvrcBNwDXhLzfdQtQn/S4jizpdjSzAmIh8C13/17Y9cRdAdxoZtcBRUC5mX3T3d8Scl0Q+7dscfdEy+kBYkEQtlcBz7l7G4CZfQ94KTDtIMjFrqHDwMviX78S2BtiLcl+QKwezOwcoJAsWPnQ3Xe4+3x3b3D3BmK/JJdkIgQmYmbXAn8F3OjuPSGXswVYbWbLzayQ2CDeppBrwmLp/WVgt7v/U9j1JLj7h9y9Lv4zdTPw8ywJAeI/281mdm780DXArhBLSjgIvNjMSuL/rteQpkHsWdkimMA7gM/EB1v6gNtCrifhHuAeM3saGADeFvI73Jngc8Ac4L/irZXH3P32MApx9yEzuwN4iNhsjnvcfWcYtYxwBXArsMPMnowf++v4fuIytncB34qH+j7gj0Kuh3g31QPAE8S6QreRpqUmtMSEiEiOy8WuIRERSaIgEBHJcQoCEZEcpyAQEclxCgIRkRynIBARyXEKAhGRHKcgEJkmM7ssvidCkZmVxteLXxd2XSKp0g1lImlgZn9PbM2cYmLr1Hwi5JJEUqYgEEmD+FIEW4gtW/JSdx8OuSSRlKlrSCQ9qoEyYC6xloHIjKEWgUgamNkmYruSLQcWufsdIZckkrJcXH1UJK3M7K3AkLt/O7538a/M7JXu/vOwaxNJhVoEIiI5TmMEIiI5TkEgIpLjFAQiIjlOQSAikuMUBCIiOU5BICKS4xQEIiI57v8DtrmKKSA9rTgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "x.grad.zero_()\n",
    "y.sum().backward()\n",
    "xyplot(x, x.grad, 'grad of tanh')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_08ekj8u",
    "id": "646F897331DD48FE9391B95A0B843BFB",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### 关于激活函数的选择\n",
    "\n",
    "ReLu函数是一个通用的激活函数，目前在大多数情况下使用。但是，ReLU函数只能在隐藏层中使用。\n",
    "\n",
    "用于分类器时，sigmoid函数及其组合通常效果更好。由于梯度消失问题，有时要避免使用sigmoid和tanh函数。  \n",
    "\n",
    "在神经网络层数较多的时候，最好使用ReLu函数，ReLu函数比较简单计算量少，而sigmoid和tanh函数计算量大很多。\n",
    "\n",
    "在选择激活函数的时候可以先选用ReLu函数如果效果不理想可以尝试其他激活函数。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_o64hzes",
    "id": "1E9A9AB460B24ECA8093296483C8114A",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### 多层感知机\n",
    "多层感知机就是含有至少一个隐藏层的由全连接层组成的神经网络，且每个隐藏层的输出通过激活函数进行变换。多层感知机的层数和各隐藏层中隐藏单元个数都是超参数。以单隐藏层为例并沿用本节之前定义的符号，多层感知机按以下方式计算输出：\n",
    "\n",
    "\n",
    "$$\n",
    " \\begin{aligned} \\boldsymbol{H} &= \\phi(\\boldsymbol{X} \\boldsymbol{W}_h + \\boldsymbol{b}_h),\\\\ \\boldsymbol{O} &= \\boldsymbol{H} \\boldsymbol{W}_o + \\boldsymbol{b}_o, \\end{aligned} \n",
    "$$\n",
    "\n",
    "\n",
    "其中$\\phi$表示激活函数。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_ie66nbg",
    "id": "FFFB468B5E804BBE96DA88180CAF5E7A",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## 2、多层感知机从零开始的实现"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "graffitiCellId": "id_6a7tvf7",
    "id": "78607977A6E44F26AE3FCB420BFB8692",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.4.0\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import numpy as np\n",
    "import sys\n",
    "sys.path.append(\"..\")\n",
    "# sys.path.append(\"/home/kesci/input\")\n",
    "import d2lzh1981 as d2l\n",
    "print(torch.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_hw0exiu",
    "id": "FDA9B8A8D96C4B1E8408D64E257839D3",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### 2.1获取训练集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "graffitiCellId": "id_6ko1fuu",
    "id": "62706F910EA649E2AF9E51DD0579239B",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "batch_size = 256\n",
    "train_iter, test_iter = d2l.load_data_fashion_mnist(batch_size,root='D:\\Anaconda3\\Anaconda3_jupyter_documents\\动手学深度学习PyTorch\\FashionMNIST2065')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_oecehxg",
    "id": "EE705582E4694D16B00D335CDF519745",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### 2.2定义模型参数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "graffitiCellId": "id_a3vvut6",
    "id": "48A8DC61D8194E3890FDCF01C10C07AB",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "num_inputs, num_outputs, num_hiddens = 784, 10, 256\n",
    "\n",
    "W1 = torch.tensor(np.random.normal(0, 0.01, (num_inputs, num_hiddens)), dtype=torch.float)\n",
    "b1 = torch.zeros(num_hiddens, dtype=torch.float)\n",
    "W2 = torch.tensor(np.random.normal(0, 0.01, (num_hiddens, num_outputs)), dtype=torch.float)\n",
    "b2 = torch.zeros(num_outputs, dtype=torch.float)\n",
    "\n",
    "params = [W1, b1, W2, b2]\n",
    "for param in params:\n",
    "    param.requires_grad_(requires_grad=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_1zqiu62",
    "id": "2F74A4FFA1854D0FA21421FB95C19AA4",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### 2.3定义激活函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "graffitiCellId": "id_2mvw62w",
    "id": "314B5039D00444F1A141936DA9DEEFC7",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def relu(X):\n",
    "    return torch.max(input=X, other=torch.tensor(0.0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_5x2qauv",
    "id": "C7D87401F57A4D38875991E3FB9D8232",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### 2.4定义网络"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "graffitiCellId": "id_pxbcyd5",
    "id": "EC0C168D06FD42BA88530E69F21654FF",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def net(X):\n",
    "    X = X.view((-1, num_inputs))\n",
    "    H = relu(torch.matmul(X, W1) + b1)\n",
    "    return torch.matmul(H, W2) + b2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_pq3xcjp",
    "id": "878D7335FBF84B2F972221710AF3E85B",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### 2.5定义损失函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "graffitiCellId": "id_ep9jbkk",
    "id": "228E4F03614B4D4F8FD4136585679D30",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "loss = torch.nn.CrossEntropyLoss()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_rszqefs",
    "id": "C50B2C4AAE0248F48C1033F133FEDCFF",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### 最后，进行训练"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "graffitiCellId": "id_zzbl9up",
    "id": "35447C7516C14503951036182938A66C",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 1, loss 0.0030, train acc 0.715, test acc 0.785\n",
      "epoch 2, loss 0.0019, train acc 0.821, test acc 0.827\n",
      "epoch 3, loss 0.0017, train acc 0.843, test acc 0.838\n",
      "epoch 4, loss 0.0015, train acc 0.856, test acc 0.818\n",
      "epoch 5, loss 0.0015, train acc 0.863, test acc 0.852\n"
     ]
    }
   ],
   "source": [
    "num_epochs, lr = 5, 100.0\n",
    "# def train_ch3(net, train_iter, test_iter, loss, num_epochs, batch_size,\n",
    "#               params=None, lr=None, optimizer=None):\n",
    "#     for epoch in range(num_epochs):\n",
    "#         train_l_sum, train_acc_sum, n = 0.0, 0.0, 0\n",
    "#         for X, y in train_iter:\n",
    "#             y_hat = net(X)\n",
    "#             l = loss(y_hat, y).sum()\n",
    "#             \n",
    "#             # 梯度清零\n",
    "#             if optimizer is not None:\n",
    "#                 optimizer.zero_grad()\n",
    "#             elif params is not None and params[0].grad is not None:\n",
    "#                 for param in params:\n",
    "#                     param.grad.data.zero_()\n",
    "#            \n",
    "#             l.backward()\n",
    "#             if optimizer is None:\n",
    "#                 d2l.sgd(params, lr, batch_size)\n",
    "#             else:\n",
    "#                 optimizer.step()  # “softmax回归的简洁实现”一节将用到\n",
    "#                          \n",
    "#             train_l_sum += l.item()\n",
    "#             train_acc_sum += (y_hat.argmax(dim=1) == y).sum().item()\n",
    "#             n += y.shape[0]\n",
    "#         test_acc = evaluate_accuracy(test_iter, net)\n",
    "#         print('epoch %d, loss %.4f, train acc %.3f, test acc %.3f'\n",
    "#               % (epoch + 1, train_l_sum / n, train_acc_sum / n, test_acc))\n",
    "\n",
    "d2l.train_ch3(net, train_iter, test_iter, loss, num_epochs, batch_size, params, lr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_vlyl5rw",
    "id": "BA81101ED80E4F12982C8145D848F8F3",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## 3、多层感知机pytorch实现"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "graffitiCellId": "id_y3ww285",
    "id": "717B8BFCFA264BF18F5A0A4972F73194",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.4.0\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "from torch import nn\n",
    "from torch.nn import init\n",
    "import numpy as np\n",
    "import sys\n",
    "sys.path.append(\"..\")\n",
    "# sys.path.append(\"/home/kesci/input\")\n",
    "import d2lzh1981 as d2l\n",
    "\n",
    "print(torch.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_789ngth",
    "id": "4D9F4F5F30D242FB93922254452D79C2",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### 3.1初始化模型和各个参数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "graffitiCellId": "id_r2zss0i",
    "id": "5B2A503B8DF6469698CC54C189494FAA",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "num_inputs, num_outputs, num_hiddens = 784, 10, 256\n",
    "    \n",
    "net = nn.Sequential(\n",
    "        d2l.FlattenLayer(),\n",
    "        nn.Linear(num_inputs, num_hiddens),\n",
    "        nn.ReLU(),\n",
    "        nn.Linear(num_hiddens, num_outputs), \n",
    "        )\n",
    "    \n",
    "for params in net.parameters():\n",
    "    init.normal_(params, mean=0, std=0.01)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_6lb8nps",
    "id": "FA88F8D978D14AC082678D8B5C7E2AA5",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### 最后，进行训练"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "graffitiCellId": "id_0az2fbu",
    "id": "505BC0B5184046608BEB4A54C249172C",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 1, loss 0.0031, train acc 0.704, test acc 0.791\n",
      "epoch 2, loss 0.0019, train acc 0.821, test acc 0.784\n",
      "epoch 3, loss 0.0016, train acc 0.847, test acc 0.802\n",
      "epoch 4, loss 0.0015, train acc 0.856, test acc 0.826\n",
      "epoch 5, loss 0.0015, train acc 0.862, test acc 0.822\n"
     ]
    }
   ],
   "source": [
    "batch_size = 256\n",
    "train_iter, test_iter = d2l.load_data_fashion_mnist(batch_size,root='D:\\Anaconda3\\Anaconda3_jupyter_documents\\动手学深度学习PyTorch\\FashionMNIST2065')\n",
    "loss = torch.nn.CrossEntropyLoss()\n",
    "\n",
    "optimizer = torch.optim.SGD(net.parameters(), lr=0.5)\n",
    "\n",
    "num_epochs = 5\n",
    "d2l.train_ch3(net, train_iter, test_iter, loss, num_epochs, batch_size, None, None, optimizer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用pytorch实现简便很多.."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
